#110 tail-sampling 时间轮内存优化:压缩入轮 + span 谓词标记 + 决策免解压 + 大包分级落盘 - #206
Merged
Conversation
改动链路: - DataPacket 新增 payload_compression(zstd 压缩态存储,小包无收益回退) - PickTrace 组包后压缩;WalkRawPBPoints/合并路径按标记解压 - span 谓词摘要(pred_error/http/biz/trace_keep + duration 摘要),Ingest 合并 OR 聚合(跨 Datakit) - evaluatePipelines 快速路径:condition 静态编译为谓词判定(filter 语义严格等价,未覆盖回退 walk);谓词全零(旧数据)回退 - 时间轮大包分级落盘:PayloadSpiller 接口 + 文件实现,惰性读回,决策后释放 风险点:谓词为已见 span 摘要语义;损坏 payload 首点预检保持原语义;空谓词包回退 walk。 测试:aggregate 全量 ok;新增 zstd/predicates/pipeline_predicates/spill 四组测试;lint 通过。
旧版本 Datakit(未计算 span 谓词)的数据由 dataway 入轮前补算, 使决策快速路径对任意客户端版本生效;新版 Datakit 数据(谓词非全零) 不重算,升级完成后补算路径零触发。谓词提取重构为统一 getter, Point(组包)与 PBPoint(补算)两条路径共用同一逻辑。
1. Ingest 合并路径:hydrate(读回 spill payload)失败时跳过合并, 避免 payload 只有新包而 PointCount 累计造成数据错位与决策错误。 2. ComputeSpanPredicates:先计算到临时结构,任一 span 解码失败则整体 返回错误且不写入部分谓词,谓词保持全零回退解压 walk。 附两个回归测试(含能捕获旧缺陷的构造)。
songlonqi-java
force-pushed
the
iss-110-tailsampling-fix
branch
from
August 13, 2026 03:25
8f26dab to
5909728
Compare
issue #110 涉及的代码注释全部英文化(zstd 封装、span 谓词、决策快速 路径、时间轮落盘、proto 字段说明、测试文件),保证开源仓库注释语言一致。 风险点:纯注释/文档变更;proto 注释变更后已重新生成 pb.go。 测试:go test ./aggregate/ ok;lint 无新增问题。
leecha
self-requested a review
August 13, 2026 07:27
leecha
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
tail-sampling 时间轮按"全量展开数据 × TTL"常驻内存。按客户实测(压缩 12TB/天、展开 170TB/天):TTL=30s 驻留 ~35GB,TTL=60s ~71GB,单机内存的主要压力源。
改动(aggregate 包)
1. payload 压缩入轮
DataPacket新增payload_compression(0=raw / 1=zstd),PickTrace组包后压缩,小包无收益自动回退WalkRawPBPoints/ 合并路径按标记解压;时间轮驻留从展开态降为压缩态(~10-17x)2. span 谓词标记(跨 Datakit OR 合并)
DataPacket新增谓词字段:pred_error/pred_http_error/pred_biz_error/pred_trace_keep/max_span_duration_us/root_duration_us/max_nonroot_duration_usPickTrace组包时提取(缺失字段不命中,与 filter 语义严格一致,逐 case 等价测试);Ingest 合并路径 OR 聚合(跨 datakit 任一命中即命中)ComputeSpanPredicates:dataway 对旧版 datakit 数据补算,解除版本绑定3. 决策免解压
4. 大包分级落盘
PayloadSpiller接口 +FilePayloadSpiller(打开时清空崩溃残留、key 字符集防路径穿越)GlobalSampler.SetPayloadSpiller:超阈值 payload 落盘,时间轮只存元数据;决策惰性读回(谓词可覆盖时零读盘),决策后释放测试
go test ./aggregate/:okgolangci-lint run ./aggregate/...:新增代码 0 issues风险点