export buildkit history and structured lifecycle telemetry at teardown - #128
export buildkit history and structured lifecycle telemetry at teardown#128piob-io wants to merge 1 commit into
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
5ce4997 to
303b922
Compare
| // Export this job's build history (raw record + trace bytes) and delete | ||
| // the records afterwards so history.db and its content-store attachments | ||
| // never get committed to the sticky disk. Must happen before shutdown. | ||
| const builds = await exportBuildHistory(buildkitdAddr, lifecycle); |
There was a problem hiding this comment.
is this valid? The sticky disk as a block io device doesn't know about which logical files have been deleted, right?
There was a problem hiding this comment.
Right — the block device has no notion of file deletion, so residual bytes can remain in freed blocks after commit; we don't claim physical scrubbing. What the delete does guarantee is filesystem-level: it happens via buildkitd's UpdateBuildHistory{Delete} API (BoltDB record delete + content-store blob unref) before buildkitd shutdown → integrity check → unmount → commit, so the committed filesystem no longer references those records. The purpose is (1) history.db doesn't grow unboundedly across runs and (2) the next job's export doesn't re-export prior jobs' records — not data erasure. I've reworded the comment to say exactly that (was misleadingly phrased as "never get committed").
303b922 to
87c7d43
Compare
|
@devin do we bound the size of the report we generate in terms of bytes? I think we need a way to hard bound it somewhere around 10s of MiB and under 128 MiB |
|
Yes — the report is hard-bounded to ~10 MiB worst case, enforced at assembly time in
Everything else in the report (lifecycle facts, du totals, per-cache-mount rows) is small fixed-size fields, so the whole |
At post-step teardown, export the job's raw BuildKit history records (plus solve-status traces, logs excluded, size-capped) via ListenBuildHistory, delete them so history.db and its content-store attachments are never committed to the sticky disk, and ship them with the runner _diag step timeline and structured lifecycle facts (builder mode/fallback reason, commit decision/skip reason, integrity outcome, buildctl du totals incl. per-cache-mount, fs usage) to the vm-agent's new ReportDockerBuild RPC. docker_build_ids are issued host-side; all reporting is fail-soft with bounded timeouts and never fails the customer job. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
87c7d43 to
b357178
Compare
2 participants
Summary
Guest half of the docker-builds observability design: at post-step teardown the action exports the job's raw BuildKit history and ships it, with the runner step timeline and structured lifecycle facts, to the vm-agent's new
ReportDockerBuildRPC on the existing StickyDiskService channel.src/build-telemetry.ts):exportBuildHistorystreamsControl.ListenBuildHistory(EarlyExit) against the local buildkitd, reads eachBuildHistoryRecordas a typed protobuf-es message (Ref,CompletedAt,tracedescriptor), clears the logs descriptor, and re-serializes withtoBinary— protobuf-es retains unknown fields across the round trip, so record fields newer than the vendored proto still reach the vm-agent intact. It then fetches the solve-status trace attachment from buildkitd's containerd content service and applies per-record/per-trace/total payload caps. In-flight builds are exportedincomplete; oversized ones are markedtruncated.pruneBuildHistorythen deletes the exported refs viaUpdateBuildHistory{Delete}so history.db and its content-store blobs never get committed to the sticky disk.buildctl du -vtotals incl. a per-cache-mountNested(mount_id, bytes, records)snapshot, fs used/size, prune bytes, hotload/ready/shutdown durations, sigkill flag._diaglog ships raw (size-capped); parsing and build↔step attribution happen agent-side.expose_id;docker_build_ids come back in the fa response. All reporting is fail-soft with bounded timeouts (15s export, 10s report) — telemetry can never fail the customer job.Protos live under
proto/: the fa stickydisk proto plus BuildKit's control API and containerd's content API vendored verbatim from pinned upstream releases (buildkit v0.32.2, containerd v2.3.4) viascripts/fetch-protos.sh— no hand-maintained proto subsets and no hand-written wire-format parsing. TypeScript bindings are generated intosrc/gen/at dev time viapnpm generate(buf + protoc-gen-es); nothing is fetched or compiled at action run time.dist/is rebuilt.Paired PRs: FastActions/fa (agent-side parsing/ingest) and useblacksmith/web (ClickHouse migration).
Link to Devin session: https://app.devin.ai/sessions/0c6ebc2a6de54933bebe821e21ab8b76
Requested by: @piob-io
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.