Implement new Giga GarbageCollector interface - #3868
Conversation
PR SummaryMedium Risk Overview The collector still takes the minimum head and pruning boundaries each cycle, but BlockDB: ReceiptDB (littidx): GC adapter, FlatKV: State WAL: GC surface with atomic Config/golden/test updates only; wiring the collector in production is explicitly out of scope (defaults leave Reviewed by Cursor Bugbot for commit 7ccdf29. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Beyond the inline finding, I also checked two other candidate issues: the BlockDB litt table rename from "ledger" to "blocks" (no live production data exists under the old table name for this not-yet-deployed Giga store, so this isn't an on-disk compatibility break), and littidx ReceiptDB losing its background pruner with no ExternalPruning fallback (expected per this PR's description — the collector wiring lands in a follow-up PR, consistent with this repo's staged-rollout convention).
Extended reasoning...
This PR is large and touches critical, not-yet-wired pruning logic across four Giga storage stores. One nit-level bug was already found and posted inline (statewal.New skips config.Validate). Beyond that, I reviewed two additional candidate concerns raised by finder agents and ruled both out: the BlockDB table rename is safe because the Giga block store has no production data to migrate yet, and the ReceiptDB pruner removal is explicitly called out in the PR description as intentional pending the follow-up collector-wiring PR.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3868 +/- ##
==========================================
- Coverage 61.61% 61.01% -0.60%
==========================================
Files 2348 2279 -69
Lines 200852 193282 -7570
==========================================
- Hits 123755 117932 -5823
+ Misses 66044 64852 -1192
+ Partials 11053 10498 -555
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Beyond the inline finding, I also checked the littidx TTL depth under ExternalPruning: the litt TTL is set from KeepRecent independent of the collector's RollbackWindow, so it could in principle expire receipt bodies before the collector's retention floor — but with no StorageGarbageCollector constructed anywhere yet and ExternalPruning unreachable from app.toml, this can't produce a live gap today. Not a blocker.
Extended reasoning...
Checked whether littidx's TTL (derived from KeepRecent, littTTLPerBlock) could expire receipt bodies earlier than a StorageGarbageCollector's RollbackWindow would require once ExternalPruning is wired up. Confirmed this is currently unreachable: ExternalPruning is mapstructure:"-" (no app.toml key) and no code in this PR or elsewhere constructs a StorageGarbageCollector, so the mismatch has no live path today. Recording this so a future PR that wires up the collector re-checks the TTL depth against RollbackWindow before enabling ExternalPruning in production.
| func (s *littReceiptStore) GetRetentionWindow() int64 { | ||
| if s.keepRecent <= 0 { | ||
| return gc.InfiniteRetentionWindow | ||
| } | ||
| return s.keepRecent | ||
| } |
There was a problem hiding this comment.
🟡 littReceiptStore's litt-table TTL (litt_receipt_store.go, set as cfg.KeepRecent * littTTLPerBlock) is derived from KeepRecent alone and ignores RollbackWindow, while under ExternalPruning the collector's visible retention floor is RollbackWindow + GetRetentionWindow() (litt_receipt_gc.go:52-57 returns keepRecent verbatim). Once a StorageGarbageCollector is wired in with a non-trivial RollbackWindow and a small KeepRecent, litt's TTL-driven segment GC can physically reclaim receipt bytes for blocks still at or above earliestVersion, causing GetReceiptFromStore to silently return not-found for receipts the collector still considers retained. Fix by deriving the TTL from RollbackWindow + KeepRecent (or an independent wall-clock failsafe, as blockDB does) rather than KeepRecent alone.
Extended reasoning...
The bug. newLittReceiptStore (litt_receipt_store.go) sets the litt table's body TTL once at construction:
if cfg.KeepRecent > 0 {
if err := receipts.SetTTL(time.Duration(cfg.KeepRecent) * littTTLPerBlock); err != nil {littTTLPerBlock is a fixed 2s/block constant. This derivation depends only on KeepRecent and is applied unconditionally regardless of cfg.ExternalPruning.
Meanwhile, this PR's littReceiptStore.GetRetentionWindow (litt_receipt_gc.go:52-57) returns keepRecent verbatim as the extra retention the StorageGarbageCollector should keep beyond the shared RollbackWindow. Combined with GetPruningBoundary's unconditional cutLine passthrough, the collector's logical retention floor once wired in is head - RollbackWindow - keepRecent (via pruneBlocksBelow advancing earliestVersion to that cut line). So the store's contract is to keep RollbackWindow + keepRecent blocks servable, but its physical reclaim mechanism (the litt TTL) only retains bodies for roughly keepRecent blocks worth of wall-clock time.
Why nothing today prevents the gap. litt's TTL-based segment GC reclaims whole segments once their age exceeds the TTL, independent of the tag-index earliestVersion floor. The existing code comment ("visible retention never exceeds that floor regardless of GC timing") only holds when the TTL window is deeper than the logical retention floor — true before this PR, when the only retention driver was KeepRecent itself. This PR adds a second, deeper retention driver (RollbackWindow) to the logical floor without adding a matching term to the TTL, breaking that invariant.
Concretely: at Giga's ~7ms block time, a KeepRecent*2s TTL covers roughly keepRecent * 285 blocks. With the default RollbackWindow = 1000 and, say, KeepRecent = 3 under ExternalPruning, the logical floor sits 1000 + 3 = 1003 blocks behind head, but the physical TTL only protects the most recent 3 * 285 ≈ 855 blocks. Blocks between roughly 855 and 1003 blocks behind head are still at/above earliestVersion (the collector still considers them retained and servable) but their bodies may already have been physically reclaimed by litt's TTL GC.
Step-by-step proof of the read failure:
- A
StorageGarbageCollectorwithRollbackWindow = 1000manages alittReceiptStoreconfigured withKeepRecent = 3andExternalPruning = true. GetRetentionWindow()returns3; each prune cycle advancesearliestVersiontohead - 1000 - 3 = head - 1003.- litt's TTL GC, running independently, reclaims segments for bodies older than
3 * 2s = 6sof wall clock — roughly855blocks behind head at Giga speed. - A client calls
GetReceiptFromStorefor a tx in blockhead - 900(still≥ earliestVersion = head - 1003, so the collector's contract says it must be servable). s.receipts.Get(txHash[:])returnsexists = falsebecause litt's TTL GC already reclaimed that segment — this check runs before thebelowRetentionFloorcheck even executes.- The method returns
ErrNotFoundfor a receipt that, per the store's own advertised retention contract, should still be present.
Fix. Derive the litt TTL from RollbackWindow + KeepRecent under ExternalPruning (or, more robustly, use an independent wall-clock failsafe decoupled from the block-count window, the way blockDB keeps Retention and RetentionWindow as fully separate knobs).
Severity note. All three independent verifiers rated this nit, and I agree: per the PR description, no code constructs a StorageGarbageCollector yet, so ExternalPruning is never actually driven by a live collector on any current path — the gap is real but currently unreachable. It also affects only auxiliary, non-consensus RPC receipt data, which already tolerates a missing body as a not-found response. It's worth fixing while this code is fresh, but it doesn't block merge.
There was a problem hiding this comment.
Well-documented, well-tested implementation of gc.PrunableStore across the four Giga stores, with the collector-side ExternalPruning plumbing looking correct. One blocking issue: the LittDB block table is silently renamed from "ledger" to "blocks", which is a persisted on-disk identifier with no migration, and is not mentioned anywhere in the PR description.
Findings: 1 blocking | 8 non-blocking | 4 posted inline
Blockers
- None at the file/PR level.
- 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- Cursor's second-opinion pass produced no output (
cursor-review.mdis empty), so this review reflects only Claude's and Codex's findings. Codex reported a single finding (the block table rename), which I independently confirmed and raised inline. sei-db/state_db/sc/flatkv/store.go:InitializeDataDirectoriesandapplyPebbleMetricsConfigare moved verbatim from the top of the file to the bottom (andvar _ Store = (*CommitStore)(nil)moved up) — ~80 lines of pure code motion with no behavior change and no relation to GC. Worth dropping to keep the diff reviewable.- No end-to-end test constructs a real
StorageGarbageCollectorover the four concrete stores: each store is tested against the interface in isolation, and the collector only against mocks. The load-bearing interaction the PR exists for — FlatKV's snapshot boundary holding the state WAL back — is therefore only covered bymockStore. Understood that wiring is deferred, but a single integration test over the real four would pin the invariant the mocks assert. - The preconditions in
store_gc.goandstate_wal_gc.go("FlatKV must be managed alongside the state WAL, and vice versa") are documented but unenforced. The converse combination is the sharper one: self-pruning FlatKV + collector-managed state WAL leavestryTruncateWALlive, and it can truncate the WAL above the collector's shared minimum, dropping the replay range SS still needs. Consider a construction-time check wherever the collector is eventually wired. statewalandflatkveach got a race-detector test for the GC surface against a concurrent writer (TestGCConcurrentWithWriter,TestGCConcurrentWithCommitter);blockDBdid not. It is fully mutex-guarded so it is likely fine, but note thatGetLatestBlockandPruneBelowboth takes.mu, the same mutexWriteBlock/WriteQChold — so the collector's goroutine can now block behind block writes. A matching parallel test would pin both the safety and the intent.- 3 suggestion(s)/nit(s) flagged inline on specific lines.
| return &BlockDBConfig{ | ||
| Litt: littConfig, | ||
| Retention: 24 * time.Hour, | ||
| RetentionWindow: 10000, |
There was a problem hiding this comment.
[suggestion] A non-zero default here is a fleet-wide retention decision, not a blockDB-local one. Because the collector prunes every store to the shared minimum (storage_garbage_collector.go calls this out explicitly: "receiptDB retention 100_000 also keeps SC/SS snapshots that far back"), defaulting RetentionWindow to 10 000 means that once wiring lands, FlatKV's snapshots, ReceiptDB and the state WAL all retain 10 000 blocks beyond RollbackWindow — purely because blockDB's default says so. Every other store added in this PR reports 0.
Suggest defaulting to 0 (matching the "no extra beyond the shared window" contract) and making 10 000 an explicit opt-in at the call site, or documenting here why 10 000 is the right floor for the whole fleet.
Related: AutobahnBlockDBConfig in sei-tendermint/config/autobahn.go grew no override for RetentionWindow even though it has one for Retention, so a tendermint-side operator gets this default with no way to tune it.
| // more snapshots than the count-based default. | ||
| // | ||
| // Default: false | ||
| ExternalPruning bool `mapstructure:"external-pruning"` |
There was a problem hiding this comment.
[suggestion] This field is mapstructure-addressable while its exact counterpart on ReceiptStoreConfig is mapstructure:"-", and the receipt-side comment gives the reason: enabling it is "only correct when this store is registered with a running StorageGarbageCollector, which is a property of how the process was wired and not something an operator can assert from app.toml. Exposing a key for it would let a node stand its pruner down with nothing to replace it, and the resulting unbounded growth is silent."
That argument applies at least as strongly here — this flag stands down two mechanisms (pruneSnapshots and tryTruncateWAL), so a stray external-pruning = true under [state-commit.flatkvconfig] leaves both snapshots and the state WAL with nothing bounding them. The toml_test.go addition keeps it out of the generated template, but mapstructure still binds a hand-written key.
Two options: tag it mapstructure:"-" to match the receipt field, or, if it is meant to stay operator-settable, pin the new read in testutil/configtest per the repo's configuration-reads guidance.
Separately: the receipt path rejects the unsupported combination at startup (newReceiptBackend errors on pebble + ExternalPruning), but nothing on the FlatKV path validates that a collector exists. Worth an equivalent guard wherever the collector gets constructed.
| } | ||
|
|
||
| var errs error | ||
| pruned := 0 |
There was a problem hiding this comment.
[nit] pruned is initialized and incremented but never read — each deletion already logs individually at line 91. Either drop it, or use it for a single summary log line at the end (which would also be cheaper than one Info per snapshot when a deep backlog is cleared).
There was a problem hiding this comment.
I didn't find any new issues in this pass, but this is a large, critical-path storage change — four different store implementations (BlockDB, ReceiptDB, StateWAL, FlatKV) are being given a shared PrunableStore/ExternalPruning contract that a follow-up StorageGarbageCollector will drive — so it's worth a human look rather than a bot approval.
What was reviewed: the ExternalPruning/GetRetentionWindow/GetPruningBoundary implementations across all four stores and the collector's cutLine/pruneHeight/self-pruning-store logic in storage_garbage_collector.go; confirmed the empty-store PruneBelow issue flagged earlier in this thread has been fixed (pruneBlocksBelow now no-ops when latestVersion <= 0); and checked FlatKV's ExternalPruning mapstructure:"-" omission — it's intentionally unexposed in app.toml like its receipt-store sibling, not a missing guard.
Extended reasoning...
This PR introduces gc.PrunableStore implementations for BlockDB, ReceiptDB, StateWAL, and FlatKV, plus the StorageGarbageCollector decision logic that will eventually drive pruning across all of them, and renames the public littblock.LittBlockConfig type (rippling into sei-tendermint). No collector is wired up yet in this PR — that is deliberately left for a follow-up — so the immediate behavioral risk is low, but the contracts being established here (ExternalPruning, GetRetentionWindow, GetPruningBoundary, the cutLine/pruneHeight minimum) are exactly what future pruning correctness will depend on.
No security-sensitive surface (auth/crypto/permissions) is touched; the risk here is data-retention correctness (accidentally pruning data a store still needs, or leaking unbounded growth), not exploitability.
Scrutiny should be high: this is core sei-db storage-lifecycle code, not app/consensus logic, but a mistake in the shared-minimum logic or in an ExternalPruning stand-down could silently either delete data needed for rollback or grow disk unbounded once the collector is wired up in the next PR. The PR is also large (29 files) and threads a subtle invariant (self-pruning stores still vote but never receive PruneBelow) through four independent implementations, which increases the chance of a one-store inconsistency.
The thread already contains substantive back-and-forth (cursor[bot] and my own prior comments) covering an empty-store PruneBelow bug (now fixed), a BlockDB table rename the author confirmed is safe (no existing data), and two still-open nits (statewal.New not calling config.Validate(), and the receipt store's litt TTL not accounting for RollbackWindow once ExternalPruning is wired up) that were assessed as non-blocking since no current code path can reach them. Given the size, the number of new invariants being introduced, and that not every open thread has been explicitly resolved, this is a good candidate for a human pass before merge.
There was a problem hiding this comment.
This PR implements a new fleet-wide StorageGarbageCollector and wires gc.PrunableStore into BlockDB, ReceiptDB, StateWAL, and FlatKV — no bugs were found this run, but given the scope and criticality of this change I think it's worth a human look before merging.
Beyond the bug hunting system's findings, this run also checked: FlatKV/state-WAL/BlockDB/ReceiptDB GetPruningBoundary/GetLatestBlock contracts for off-by-one and empty-store edge cases, the self-pruning-vs-collector ExternalPruning handoff in all four stores, and the two candidate issues the bug hunting system ruled out this run (receiptDB's boundary vs. its local pruner floor, and blockDB's ExternalPruning vs. its RetainHeight-driven pruner) — both were not real bugs. I also confirmed the concerns from earlier review rounds (the empty-store PruneBelow gap in littReceiptStore, the stale .LittBlockConfig/.BlockDBConfig comment, and the unreachable RetentionWindow validation in statewal) have since been addressed in later commits.
Extended reasoning...
Overview
This PR adds a shared StorageGarbageCollector (sei-db/management/gc) and implements gc.PrunableStore for four storage components — littblock.BlockDB, the littidx ReceiptDB, StateWAL, and FlatKV's CommitStore — so a single collector can coordinate pruning/retention across the whole Giga storage fleet instead of each store pruning independently. It also renames littblock.LittBlockConfig to BlockDBConfig (rippling into sei-tendermint) and renames the underlying litt table from ledger to blocks. None of this is wired into a running collector yet per the PR description — that is deferred to a follow-up PR.
Security risks
No injection/auth/crypto surface here. The real risk category is data-durability/consensus-adjacent correctness: a wrong pruning boundary or retention-window calculation could silently delete blocks, receipts, WAL entries, or state snapshots still needed to serve a rollback, or (in the WAL table-rename case) strand already-persisted data under an old table name. These failure modes are typically silent (unbounded disk growth or missing data discovered much later) rather than loud, which raises the value of a careful human pass even absent a concrete bug today.
Level of scrutiny
High. This is new infrastructure for coordinating retention across BlockDB, ReceiptDB, StateWAL and FlatKV — all storage paths that back consensus and RPC serving for Giga. Cursor's own summary flagged it "High Risk" for the same reason. The code is not live yet (no construction site calls NewStorageGarbageCollector), which lowers the blast radius today, but the interface contracts being established here (GetPruningBoundary, GetRetentionWindow, ExternalPruning) will govern real pruning behavior once wired up in the next PR, so getting them right now matters.
Other factors
Test coverage is extensive (a *_gc_test.go suite per store plus collector-level tests, per the PR description), and this run's independent look, plus the bug hunting system's, did not surface a live bug. Several issues raised in earlier review rounds on this PR (an empty-store PruneBelow gap in the receipt store, a stale comment, and a validation method that was never called) have since been fixed or the underlying field removed entirely, which is a good signal the author is engaging with feedback. Given the size (29 files) and the criticality of the code paths involved, I'd still like a human maintainer to sign off.
There was a problem hiding this comment.
No new bugs found this run beyond the existing inline findings on this PR. Beyond those, I checked whether CommitStore.reopenWAL/Rollback swapping in a new stateWALImpl would orphan a collector-held gc.PrunableStore reference to the old (now-closed) WAL — it would, but no code in this PR (or elsewhere in the tree) constructs a StorageGarbageCollector yet, so this is a wiring concern for the follow-up PR that actually registers these stores, not a bug in what's merged here.
Extended reasoning...
The candidate issue was: stateWALImpl's identity as a gc.PrunableStore is invalidated whenever CommitStore.reopenWAL or CommitStore.Rollback swaps s.wal for a freshly constructed statewal.New(...) instance. If a StorageGarbageCollector held a reference to the old stateWALImpl object, that reference would silently go stale (PruneBelow on it would just error 'state WAL is closed', and the real live WAL would never be pruned again). This is refuted as a bug in the current PR because no code anywhere in the tree constructs a StorageGarbageCollector with these stores yet — the PR description explicitly defers that wiring to a follow-up PR. Whoever writes that wiring will need to either re-register the store after a reopen, or expose a stable handle that survives it; that is a real design point to track at wiring time, but not something the current diff gets wrong.
| // ExternalPruning hands retention to the StorageGarbageCollector: the littidx backend | ||
| // stops running its own KeepRecent pruner and answers gc.PrunableStore.ExternalPruning | ||
| // with this value, so the collector prunes it instead. | ||
| // | ||
| // Like KeepRecent this is not read from the receipt-store config. It is set by whatever | ||
| // constructs the collector, because it is only correct when this store is actually | ||
| // registered with a running one — nothing here can check that, and the failure is silent | ||
| // and unbounded: the retention floor simply stops advancing. | ||
| // | ||
| // It is one field rather than two so the KeepRecent pruner and the collector can never | ||
| // both be enforcing retention. They would disagree, and the local pruner would win by | ||
| // deleting the rollback headroom the collector exists to preserve. | ||
| // | ||
| // Only the littidx backend honors this. The pebbledb backend is not a gc.PrunableStore, | ||
| // so the collector would not prune it and setting this would leave it with no pruner at | ||
| // all; newReceiptBackend rejects that combination rather than growing without bound. | ||
| ExternalPruning bool `mapstructure:"external-pruning"` |
There was a problem hiding this comment.
🟡 ReceiptStoreConfig.ExternalPruning is tagged mapstructure:"external-pruning", making it addressable from app.toml, but its own doc comment says it is "not read from the receipt-store config" (like KeepRecent) and the accompanying test explicitly asserts it's tagged mapstructure:"-". Please tag it mapstructure:"-" to match the documented contract and the sibling KeepRecent field.
Extended reasoning...
ReceiptStoreConfig.ExternalPruning (sei-db/config/receipt_config.go:74) is declared with mapstructure:"external-pruning", a real, addressable config key. This directly contradicts three independent statements of the intended contract that this same PR introduces:
- The field's own doc comment: "Like KeepRecent this is not read from the receipt-store config. It is set by whatever constructs the collector..."
- Its sibling field
KeepRecent, which the comment explicitly compares itself to, and which is correctly taggedmapstructure:"-". TestManifestNamesEveryFieldin receipt_config_fuzz_test.go, which excludesExternalPruningfrom key-manifest coverage with the reasoning "ExternalPruning is tagged mapstructure:"-"" — a factually false statement about the code as written.
All three signals point to the same intended tag (mapstructure:"-"), so this reads as a copy/paste or oversight rather than a deliberate choice.
Why the test doesn't catch it: CheckManifestCoversEveryField (testutil/configtest/golden.go) keys off the Go field name via spec.Path, not the actual mapstructure tag value. So the test's exclusion of ExternalPruning passes regardless of what the tag says — the prose justifying the exclusion is unchecked and has silently drifted from the code.
Current impact — none, but the guard is silently degraded. ReadReceiptConfig populates ReceiptStoreConfig field-by-field via explicit opts.Get(flag) calls, and there is no flagRSExternalPruning constant and no generic mapstructure/viper Unmarshal of this struct anywhere in the tree. So a hand-written external-pruning = true under [receipt-store] in app.toml is not read today, and no live exploit exists via the current app.toml loading path.
Why it's still worth fixing. The entire point of using mapstructure:"-" here (per the field's own comment) is defense-in-depth against any future or alternative config-binding mechanism (e.g. a generic mapstructure decode, a config migration tool, or a refactor of ReadReceiptConfig to a reflective bind) accidentally mapping an operator-supplied key onto a field that must only be set by whatever constructs the StorageGarbageCollector. Enabling ExternalPruning without a running collector attached would silently stand down the store's own KeepRecent pruner with nothing to replace it — unbounded growth with no error and no log, which is precisely the failure mode the surrounding comment (and the analogous FlatKV ExternalPruning field raised separately by seidroid[bot]) warns about. Leaving the tag as external-pruning removes that safety net for no benefit, since nothing currently relies on the key being addressable.
Step-by-step proof of the mismatch:
- Read the field declaration:
ExternalPruning boolmapstructure:"external-pruning"`` — this is a real, non-"-" tag. - Read the doc comment directly above it: "Like KeepRecent this is not read from the receipt-store config."
- Read
KeepRecent's tag a few lines up:mapstructure:"-"— confirming what "like KeepRecent" should mean for the tag. - Read the test comment in receipt_config_fuzz_test.go: "ExternalPruning is tagged mapstructure:"-"" — this statement is false for the code as written.
- Conclusion: the field's tag does not match its own documentation, its sibling's tag, or the test's stated justification. All three independently point to
mapstructure:"-"being correct.
Fix: change the tag to mapstructure:"-", matching KeepRecent and restoring the documented contract. This is a one-token change with no other code path depending on the current tag value.
| // RetentionWindow is how much history this store keeps beyond the shared rollback | ||
| // window of the StorageGarbageCollector that manages it, in blocks. It is what | ||
| // gc.PrunableStore.GetRetentionWindow answers: | ||
| // | ||
| // > 0 → that many blocks of history beyond the rollback window | ||
| // 0 → keep history to serve rollback window only | ||
| // -1 → never prune this store (gc.InfiniteRetentionWindow) | ||
| // | ||
| // Zero does NOT mean "keep everything" here, unlike the KeepRecent fields on | ||
| // StateStoreConfig and ReceiptStoreConfig, where 0 disables pruning. It is the most | ||
| // aggressive setting this field has; "keep everything" is -1. Assigning a KeepRecent | ||
| // value to this field inverts the retention it asks for. | ||
| // | ||
| // This is an input to a minimum shared across every managed store, not a policy applied | ||
| // to this store alone: a deep window here also holds back receiptDB and the SC/SS | ||
| // snapshots. Must be >= gc.InfiniteRetentionWindow. | ||
| // | ||
| // Independent of Retention, which is a wall-clock TTL failsafe underneath the watermark. | ||
| // Both must permit reclamation before any record is dropped. | ||
| RetentionWindow int64 |
There was a problem hiding this comment.
Might be useful to call out the following invariants:
Storage garbage collection guarantees the following invariants:
1. Garbage collection will not delete any data that is necessary to roll back to any block
between LatestBlock and (LatestBlock - RollbackWindow), inclusive.
2. Garbage collection will not delete block DB data that is before
(LatestBlock - RollbackWindow - RetentionWindow). This ensures that even if the
system rolls back to block (LatestBlock - RollbackWindow), it is still possible to read any
block from the last RetentionWindow blocks.
3. Garbage collection will eventually delete block data older than
(LatestBlock - RollbackWindow - RetentionWindow).
There was a problem hiding this comment.
I suggest we paste this block of invariants at each RetentionWindow or RollbackWindow config (with wording adjusted a little, the above is specific to block storage).
| // Unlike every other method here, PruneBefore may be called from a goroutine other than the WAL's | ||
| // owner, concurrently with any method including Append and Close, and implementations must support | ||
| // that without external serialization. Retention is driven by a garbage collector on its own | ||
| // goroutine, and requiring it to take the writer's turn would mean either blocking the writer or | ||
| // deferring the prune until the writer next runs — the latter stalling reclamation indefinitely on a | ||
| // WAL that has stopped receiving appends. | ||
| // | ||
| // Concurrent calls are unordered with respect to appends: whether a record appended around the same | ||
| // instant is pruned is unspecified. This costs nothing, because which records a prune actually | ||
| // reclaims is already approximate — it drops whole sealed files, and may defer the work arbitrarily. | ||
| PruneBefore(lowestIndexToKeep uint64) error |
There was a problem hiding this comment.
LGTM. Technically Flush() is also legal to call from anywhere, but it's ok for the godocs to be more restrictive than the code when it comes to threading model.
|
|
||
| var errs error | ||
| pruned := 0 | ||
| scanErr := traverseSnapshots(dir, true, func(version int64) (bool, error) { |
There was a problem hiding this comment.
The following would probably simplify this code a lot:
- Create a method
getSnapshotBlocks() ([]uint64, error)that returns a slice of block numbers in sorted order for all the snapshots on disk. - Create a method
deleteSnapshot(block uint64) errorthat deletes a single snapshot with the given block number
This lets us avoid the lambda function, and splits apart the logic for traversing the directory structure and the logic for deciding which blocks to keep and which ones to drop.
* main: test(config): complete the GetConfig read-site coverage (PLT-893) (#3870) Remove interchain swagger API and protos (#3881) fix(flatkv): preserve empty misc values and reject malformed empty node imports (#3869) fix(evm): count post-admission apply failures in dynamic base-fee gas (CON-359) (#3871) scripts: load generator for arctic-1 and atlantic-2 (#3850) Update go-releaser heading with experimental notice (#3879) fix(evmrpc): stream request-body budget charging to close slowloris gap (PLT-780) (#3836) Remove unused interchain accounts implementation (#3875) test(config): extend golden value test coverage (PLT-893) (#3861) Update v6.6 changelog in prep to cut patch release (#3876) Close temporary rootmulti store in connection types setup (#3872) Restore LCD pagination while preserving v6.6 precompile semantics (#3867)
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
There was a problem hiding this comment.
A carefully staged, unusually well-documented and well-tested implementation of gc.PrunableStore across the four Giga stores, with a genuine correctness gap: the littidx receipt TTL is still derived from KeepRecent alone, so under ExternalPruning receipt bodies inside the collector's shared RollbackWindow can expire while the retention floor still claims them servable. Also flagged: the two new ExternalPruning config fields carry live-looking mapstructure keys that contradict their own docs and a characterization-test comment.
Findings: 2 blocking | 10 non-blocking | 7 posted inline
Blockers
- Cursor's second-opinion pass produced no output (
cursor-review.mdis empty), so this synthesis merges only Claude's and Codex's findings. - 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- Nothing in the tree constructs a
StorageGarbageCollectoror sets eitherExternalPruningfield, so every path added here is unreachable in a running node today. That is stated in the PR description and is fine as staged work, but it means the interaction between the four stores is only exercised by mocks instorage_garbage_collector_test.go— there is no test where the real FlatKV, StateWAL, ReceiptDB and BlockDB vote in one cycle. Worth adding when the collector is wired, since the cross-store invariant (SC's snapshot boundary holding the WAL back) is the whole point of the design and is currently pinned only againstmockStore. flatkv/config.Config.ExternalPruninghas no equivalent ofnewReceiptBackend's "reject the combination we cannot honor" guard — the doc says so explicitly and defers it to wherever the collector is constructed. Please make sure that follow-up actually lands a "ExternalPruning set but this store is not registered with a running collector" check; the failure mode (snapshots and the state WAL both unbounded) is silent and expensive, and the flag alone stands down two mechanisms.sei-db/state_db/sc/flatkv/store.go: relocatingInitializeDataDirectories/applyPebbleMetricsConfigto the bottom of the file and movingvar _ Store = (*CommitStore)(nil)up is pure code motion unrelated to the GC work. It adds ~76 lines of diff noise and makesgit blameon those functions point at this PR. Consider dropping it or splitting it out.TestReceiptLocalPrunerAdvancesFloorWithoutCollectorpays a real 1–2s wall-clock wait against a jittered ticker with a 6srequire.Eventuallybudget. The comment justifies it andTestRunsLocalPrunercovers the decision without waiting, so this is only a note: it is the kind of test that becomes the flake on a loaded CI runner.- 6 suggestion(s)/nit(s) flagged inline on specific lines.
| // | ||
| // KeepRecent is what this store asks for either way: the pruner's window when it | ||
| // runs, the collector's retention window when it does not, and the litt TTL in | ||
| // both cases. |
There was a problem hiding this comment.
[blocker] The litt TTL is the one thing that does not follow the collector's window. (Also raised by Codex as P1.)
newLittReceiptStore still sets receipts.SetTTL(KeepRecent * littTTLPerBlock) (line 162), i.e. the body TTL covers roughly KeepRecent blocks of wall clock. But under ExternalPruning the retention the collector enforces is RollbackWindow + GetRetentionWindow() = RollbackWindow + KeepRecent blocks.
Concretely, with ExternalPruning = true, KeepRecent = 100_000 and RollbackWindow = 100_000: the collector holds the floor at head - 200_000 and every other store retains to match, so a rollback to head - 150_000 is supposed to be servable — but litt has already expired the receipt bodies for everything older than ~100_000 × 2s. GetReceiptFromStore returns ErrNotFound for blocks the retention floor says are live, which is exactly the cross-store consistency guarantee the collector exists to provide.
The type doc here says "KeepRecent is what this store asks for either way: the pruner's window when it runs, the collector's retention window when it does not, and the litt TTL in both cases" — the third clause is the bug. The TTL needs to cover RollbackWindow + KeepRecent when external pruning is on, which means RollbackWindow has to reach this constructor (or the TTL has to be disabled under ExternalPruning and reclamation left entirely to the collector).
Latent today since nothing wires a collector, but the mapping between KeepRecent, GetRetentionWindow and the TTL is defined here, so this is where it should be resolved rather than in the wiring PR.
| // Only the littidx backend honors this. The pebbledb backend is not a gc.PrunableStore, | ||
| // so the collector would not prune it and setting this would leave it with no pruner at | ||
| // all; newReceiptBackend rejects that combination rather than growing without bound. | ||
| ExternalPruning bool `mapstructure:"external-pruning"` |
There was a problem hiding this comment.
[suggestion] The doc directly above says "Like KeepRecent this is not read from the receipt-store config", but unlike KeepRecent (line 52, mapstructure:"-") this field carries a live-looking key. ReadReceiptConfig only does explicit opts.Get lookups so nothing decodes it today — but the tag is the only structural expression of the invariant, and it currently says the opposite of the prose. Suggest mapstructure:"-" to match KeepRecent; testdata/receipt-store.golden records the field either way (as it does for KeepRecent), so the change is inert for the characterization suite.
| // sitting in a config struct that configuration cannot address is exactly the kind of | ||
| // thing a replacement manager would otherwise try to map a key onto. | ||
| "KeepRecent", | ||
| // ExternalPruning is tagged mapstructure:"-" for a sharper reason than KeepRecent: it is |
There was a problem hiding this comment.
[suggestion] This comment states as fact that ExternalPruning is tagged mapstructure:"-", but receipt_config.go:74 tags it mapstructure:"external-pruning". Per testutil/configtest/AGENTS.md these manifest exclusions are the recorded contract a replacement implementation reads, so a comment that misdescribes the tag is the specific drift the suite is meant to prevent. Fix the tag (preferred) or the comment.
| // more snapshots than the count-based default. | ||
| // | ||
| // Default: false | ||
| ExternalPruning bool `mapstructure:"external-pruning"` |
There was a problem hiding this comment.
[suggestion] Same as the receipt-side field: the doc says "Keeping the field unreachable from config is what replaces that guard", but mapstructure:"external-pruning" is a real key name. GetConfig reads FlatKV via explicit v.IsSet("state-commit.flatkv.*") calls and this one is not among them, so it is unreachable in practice — but toml_test.go:61 only asserts the key is absent from the template, which would keep passing if a future viper.Unmarshal path picked the struct up. Given the doc's own point that this one flag stands down both pruneSnapshots and tryTruncateWAL, mapstructure:"-" makes the claim structural rather than incidental.
| } | ||
|
|
||
| var errs error | ||
| pruned := 0 |
There was a problem hiding this comment.
[nit] pruned is incremented at line 90 but never read — the per-snapshot logger.Info below already carries the information. Either drop the counter or use it (e.g. a single summary log with the count, which would also be quieter than one line per snapshot when RollbackWindow / SnapshotInterval is large).
| return &BlockDBConfig{ | ||
| Litt: littConfig, | ||
| Retention: 24 * time.Hour, | ||
| RetentionWindow: 10000, |
There was a problem hiding this comment.
[suggestion] This default is a fleet-wide policy in a per-store field. As the RetentionWindow doc two dozen lines up says, the value is an input to a shared minimum — so 10000 here pushes pruneHeight 10k blocks deeper for ReceiptDB, the state WAL and the SC snapshots too, not just for BlockDB. Two things worth reconsidering: (a) whether the default should be 0 and the extra depth expressed once as RollbackWindow in StorageGarbageCollectorConfig, and (b) that AutobahnBlockDBConfig.LittBlockConfig exposes Retention but not RetentionWindow, so once wired there is no way to tune this from tendermint config.
| // This value is persisted layout, not just an identifier: littdb puts a table's data at | ||
| // <root>/<tableName>/segments, so changing it makes NewBlockDB open a fresh empty table while the | ||
| // old data sits untouched under the previous name — neither served nor reclaimed. | ||
| const tableName = "blocks" |
There was a problem hiding this comment.
[suggestion] The comment you added here states the hazard precisely — a rename makes NewBlockDB open a fresh empty table while the old data sits under <root>/ledger/ "neither served nor reclaimed" — and then the rename is performed anyway, relying on "BlockDB is not deployed on any network yet" from the PR description. That holds for mainnet, but any dev/CI/devnet home directory carrying a ledger/ table silently comes up empty rather than failing, which is the worst shape for the one class of environment where it can happen.
Since the check is cheap and the comment already argues for it: os.Stat(<root>/ledger) at open and refuse to start (or log loudly) if it exists. That turns a silent empty store into a one-line operator action, and can be deleted once no such directories remain.
There was a problem hiding this comment.
Well-constructed implementation of gc.PrunableStore across BlockDB/ReceiptDB/StateWAL/FlatKV, with the ExternalPruning single-source-of-truth pattern correctly enforced at the collector (the choke point) and thorough per-store tests. I found no blocking correctness bug; the notes below are default/behavior changes that aren't called out in the PR description, plus some diff noise and deferred-wiring risks.
Findings: 0 blocking | 11 non-blocking | 4 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- Verification notes: I traced the concurrency claim added to
seiwal.WAL.PruneBefore. Both real implementations (walImpl.PruneBefore→sendToWriter,serializingWAL.PruneBefore→submit) funnel through a channel with a shutdown-priority select, so the off-goroutine carve-out is genuinely honored.walsim.legacyWALShim.PruneBeforemutates a plain counter without a lock, but it satisfieswalStore, notseiwal.WAL, so it is not bound by the new contract. Similarly confirmedCommitStore.GetLatestBlock'sRLockmatchesCommitwritingcommittedVersionunders.mu.Lock()(store_write.go:46,116), and thatearliestVersionin the receipt store is monotonic in every writer, which is what the newgcFiltermonotonicity requirement depends on. - Stale comment:
litt_block_db.go:397still reads "gcFilter marks a key in the shared ledger table" after the table rename toblocks. Not on a changed line, so easy to miss. - Nothing in the tree constructs a
StorageGarbageCollectorover these stores (GigaStorageConfig.PruningConfigis still unwired), so every newExternalPruningstand-down path ships dark —pruneSnapshots,tryTruncateWAL, andstartPruningall keep running in every real deployment. That is explicitly the stated scope, but it means the four-store end-to-end cycle has no coverage beyond mocks. Worth a follow-up integration test at the wiring PR. AutobahnBlockDBConfig(sei-tendermint/config/autobahn.go) exposesRetentionandGCPeriodbut no override for the newRetentionWindow, so sei-tendermint-configured block DBs are pinned to the 10000 default. Harmless today (no collector), but the knob will be needed at wiring time.TestReceiptLocalPrunerAdvancesFloorWithoutCollectorspends up to 6 real seconds waiting on a 1–2s jittered ticker (~3 ticks of headroom). Under-race+ coverage on a loaded CI shard that is a plausible flake source; the comment acknowledges the wait is deliberate, but consider making the prune interval injectable so the wait can shrink.refuseLegacyTableonly distinguishes exists / not-exists; a file namedledgerin a root would produce the "pre-rename table" error even though it is not a table directory. Cosmetic, and the operator action (move it aside) is the same.- Second-opinion passes: Codex reported no material issues.
cursor-review.mdis empty — that pass produced no output, so it contributed nothing to this synthesis. - 4 suggestion(s)/nit(s) flagged inline on specific lines.
| Retention: 24 * time.Hour, | ||
| return &BlockDBConfig{ | ||
| Litt: littConfig, | ||
| RetentionTime: time.Hour, |
There was a problem hiding this comment.
[suggestion] This silently drops the TTL failsafe from 24h to 1h (the setup_test.go assertion is updated to match, so it is deliberate), but the PR description doesn't mention it among the BlockDB changes.
RetentionTime is documented right above as the guard that "even an over-eager watermark cannot delete data younger than" — shrinking it 24× shrinks exactly that safety margin, and it applies today to autobahn/devnet block DBs that have no collector at all. Please call it out in the description, or keep 24h until the collector actually owns the watermark.
| Backend: "pebbledb", | ||
| AsyncWriteBuffer: DefaultSSAsyncBuffer, | ||
| KeepRecent: 0, | ||
| KeepRecent: DefaultReceiptKeepRecent, |
There was a problem hiding this comment.
[suggestion] I confirmed the safety claim for seid: readReceiptStoreConfig (app/receipt_store_config.go:27) overwrites KeepRecent unconditionally from min-retain-blocks, and KeepRecent is mapstructure:"-" so the app.toml template is unaffected. So no node changes behavior here.
The one caller worth flagging is DefaultGigaStorageConfig (sei-db/config/giga_config.go:58), which now hands a receipt store KeepRecent = 10000 instead of "keep everything". It has no production caller today, but it is the Giga wiring path this PR is building toward — a node constructed from it would serve eth_getTransactionReceipt for only ~10k blocks, and nothing in that path re-derives KeepRecent from min-retain-blocks. Consider setting it explicitly in DefaultGigaStorageConfig (or pinning it in giga_config_test.go) so the wiring PR can't inherit this default by accident.
| // more snapshots than the count-based default. | ||
| // | ||
| // Default: false | ||
| ExternalPruning bool `mapstructure:"-"` |
There was a problem hiding this comment.
[suggestion] The doc is candid that there is no guard here and that the collector-exists check "belongs wherever the collector is eventually constructed" — agreed, but note this is the one place in the PR where the invariant is documentation only, and it is the highest-stakes one: this single flag stands down both pruneSnapshots and tryTruncateWAL, and the failure mode (nothing bounds snapshots or the state WAL) is silent and unbounded.
Per AGENTS.md's "guard at the choke point," the cleanest shape at wiring time is for NewStorageGarbageCollector to be the only thing that can turn this on — e.g. the collector constructor sets ExternalPruning on the configs it takes ownership of, rather than accepting configs that already claim it. Worth recording as the intended follow-up so the next author doesn't add a second setting that can disagree.
| c.MiscDBConfig.EnableReadWriteMetrics = c.EnableReadWriteMetrics | ||
| c.MetadataDBConfig.EnableReadWriteMetrics = c.EnableReadWriteMetrics | ||
| } | ||
| var _ Store = (*CommitStore)(nil) |
There was a problem hiding this comment.
[nit] Relocating InitializeDataDirectories / applyPebbleMetricsConfig from the top of the file to the bottom (and moving this interface assertion up in their place) is ~80 lines of pure churn unrelated to the GC work, and it makes this file's diff read as if it changed substantively when it didn't. Behavior is identical, so this is fine to keep — but a separate commit, or leaving it out, would make the PR easier to review.
Superseded: latest AI review found no blocking issues.
Superseded: latest AI review found no blocking issues.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7ccdf29. Configure here.
| default: | ||
| return gc.CannotServeRollback | ||
| } | ||
| } |
There was a problem hiding this comment.
Self-pruning FlatKV strands older snapshots
Medium Severity
With ExternalPruning left false (the default), FlatKV still votes via GetPruningBoundary, which returns only the newest snapshot at or below cutLine. StateWAL always accepts collector prunes, so the WAL can be cut to that newer snapshot while SnapshotKeepRecent still keeps older ones. Those older snapshots stay on disk but can no longer replay, so rollback to them fails. The hybrid path this PR documents as safe therefore breaks SnapshotKeepRecent depth as soon as a collector is wired without flipping the flag.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 7ccdf29. Configure here.
There was a problem hiding this comment.
Solid, unusually well-documented change: all four Giga stores implement gc.PrunableStore, the ExternalPruning flag genuinely makes "collector prunes me" and "my own pruner is off" a single fact, and the new GCFilter correctly makes the retention floor a precondition for litt reclamation. No blocking correctness or security issues found; the notes below are a coverage gap, a forward-looking guard gap, and some default/doc drift.
Findings: 0 blocking | 7 non-blocking | 3 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- Cursor's second-opinion pass (
cursor-review.md) is empty — that review produced no output. Codex (codex-review.md) reported no material issues, which matches my own read. sei-tendermint/config/autobahn.go:51still documentsAbsent ⇒ littblock.DefaultConfig Retention; the field it resolves to is nowRetentionTime. Outside the diff hunk, so noting it here.- Wiring-PR hazard worth recording now: the collector holds a specific store instance, but
CommitStore.reopenWAL()(sei-db/state_db/sc/flatkv/store.go:852) replacess.walwith a freshly openedstatewal.StateWALon the state-sync/import path. A collector registered against the oldstateWALImplwould then be pruning a closed WAL (returning errors every cycle) while the live one grows unbounded. ThePrunableStoredoc onstateWALImplstates the SC/SS co-management precondition but not this instance-identity one. - Verified the claims I could check independently:
readReceiptStoreConfig(app/receipt_store_config.go:27) does overwriteKeepRecentunconditionally, andDefaultGigaStorageConfighas no non-test callers — so theDefaultReceiptKeepRecent0 → 10000 change really is confined to tools/tests as the description says. The tx-hash read path (GetReceiptFromStore) does enforcebelowRetentionFloor, so shortening the litt TTL to a flat hour changes only physical reclamation, not visible retention. - 3 suggestion(s)/nit(s) flagged inline on specific lines.
| // its own pruner is the only one there is. Honoring ExternalPruning here would stop that | ||
| // pruner and put nothing in its place, so refuse at startup instead: unbounded receipt | ||
| // growth is not something to discover from a full disk weeks later. | ||
| if config.ExternalPruning { |
There was a problem hiding this comment.
[suggestion] This guard closes the pebbledb hole, but the littidx path can still reach the same silent-unbounded-growth state the guard exists to prevent: with ExternalPruning = true and KeepRecent == 0, runsLocalPruner() is false and GetRetentionWindow() folds to InfiniteRetentionWindow, so getCutLine returns 0, the store is never asked for a boundary, and never receives PruneBelow. Nothing prunes it.
That isn't an exotic combination — KeepRecent comes from min-retain-blocks, which defaults to 0, so it's the shape a wiring PR gets unless it remembers to set both. Not a live bug (nothing sets ExternalPruning yet), but it's exactly the combination the design section says should be unrepresentable rather than merely discouraged, and this switch is the choke point every receipt store passes through.
Suggest either rejecting ExternalPruning && KeepRecent <= 0 here alongside the pebble case, or — if "external pruning + keep everything" is meant to be legal — logging it once at open so an operator can tell it apart from a store whose floor has simply not moved yet.
| // | ||
| // Disabled by config.ExternalPruning, under which the WAL is a managed store in its own right and the | ||
| // collector prunes it to a floor derived from every store, not just from this one's snapshots. | ||
| func (s *CommitStore) tryTruncateWAL() { |
There was a problem hiding this comment.
[suggestion] TestGCExternalPruningStandsDownSnapshotPruner pins the pruneSnapshots half of this flag, but nothing pins this half. Per the ExternalPruning doc in config/config.go, this is the higher-stakes of the two — dropping the || s.config.ExternalPruning here (or reordering the nil check) leaves the state WAL with both truncators off if the collector isn't registered, and the failure is silent and expensive.
A table-driven test in the shape of the snapshot-pruner one would do it: two CommitStores differing only in ExternalPruning, a WAL with a known stored range, tryTruncateWAL(), then assert the range moved in one case and not the other. snapshot_test.go:1510 already exercises tryTruncateWAL directly, so the harness exists.
| Retention: 24 * time.Hour, | ||
| return &BlockDBConfig{ | ||
| Litt: littConfig, | ||
| RetentionTime: time.Hour, |
There was a problem hiding this comment.
[nit] Worth calling out that this is a live default change, not just a rename: AutobahnBlockDBConfig.Retention is optional, so any autobahn node without an explicit retention in its config file goes from a 24h TTL failsafe to 1h. Benign as far as I can tell (the watermark is the visible-retention gate and reads below it are already refused, so this only makes reclamation of already-released data 24× more prompt), but the PR description presents the TTL rework under the ReceiptDB heading and this one reads as incidental to the rename.
Separately: RetentionWindow: 10000 on the next line has no path from AutobahnBlockDBConfig — the only two overrides are retention and gc_period — so whenever BlockDB is registered with a collector, autobahn nodes will be stuck on the hardcoded window until that config grows a key for it.


Summary
Retention for Giga's storage components is currently decided by each store's own
pruner, independently and with no knowledge of the others. This PR implements
gc.PrunableStorefor all four of them so a singleStorageGarbageCollectorcan manage the pruning logic for the whole fleet.
Each cycle the collector asks every store for its ingest height and for the
oldest block it must keep in order to serve the shared
RollbackWindow, takesthe minimum of those answers, and prunes every store to it. The stores therefore
stay mutually consistent — a rollback target that one store can serve is one they
can all serve, which per-store pruners could not guarantee.
What's included
BlockDBlitt_block_gc.go; adds aRetentionWindowconfig fieldReceiptDBlitt_receipt_gc.go; adds anExternalPruningconfig field that gates the existing background pruner; adds a littGCFilterso reclamation follows the retention floorStateWALstate_wal_gc.go;PruneBelowcallsseiwaldirectlyFlatKV(SC)store_gc.go; snapshot-aware pruning boundary; adds anExternalPruningconfig fieldNotable design points
ExternalPruning() boolonPrunableStoreTwo stores keep a pruner of their own — FlatKV (
SnapshotKeepRecent) andReceiptDB (
KeepRecent) — because both still run without a collector: FlatKV inthe seidb tools and bench paths, ReceiptDB on any node with
rs-backend = "littidx". Both pruners active at once is unsafe: the local onewould delete the very data the collector is holding to serve the rollback window.
Standing the local one down with nothing to replace it is equally unsafe, and
fails silently — the retention floor simply stops advancing.
ExternalPruningmakes both combinations unrepresentable rather than merelydiscouraged. Each store answers it from the same
config.ExternalPruningfieldits own pruner reads to stand down, so "the collector prunes this store" and "the
store does not prune itself" are one fact instead of two settings that can
disagree. Stores with no pruner of their own return
trueunconditionally.A store that reports
falseis still asked for its boundary and still holds theshared minimum down — it just never receives
PruneBelow. Dropping it from thevote instead would prune the WAL out from under the snapshots it replays from.
Neither
ExternalPruningfield is reachable fromapp.toml(both aremapstructure:"-"). Enabling one is only correct when the store is registeredwith a running collector, which is a property of how the process was wired and
not something an operator can assert. Where it is checkable it is checked:
newReceiptBackendrejectspebbledb+ExternalPruning, because that backendis not a
gc.PrunableStoreand would end up with no pruner at all.The retention floor gates reclamation, and the TTL is only an age failsafe
Both litt-backed stores now pair their TTL with a
GCFilter, so a record isreclaimed only when it is both below the store's retention floor and
older than the TTL. Previously ReceiptDB had no filter, which made the TTL the
sole reclamation mechanism, and it was sized as
KeepRecent × 2s. UnderExternalPruningthe enforced retention isRollbackWindow + KeepRecent, so aTTL sized for
KeepRecentalone expired receipt bodies for blocks the collectorstill considered live —
ErrNotFoundinside the rollback window, which is theexact cross-store guarantee the collector exists to provide.
With the filter in place the TTL no longer has to know how many blocks anything
is, so
littTTLPerBlockis gone and both stores take a flat duration(
RetentionTime/littRetentionTime), defaulting to 1 hour. Visible retentionfollows the floor and reclamation can no longer lead it.
Retention semantics
With
R= a store'sGetRetentionWindowandF = LatestBlock - RollbackWindow - R,collection guarantees, per managed store:
[LatestBlock - RollbackWindow, LatestBlock]is deleted.Fis deleted — so even after rolling back toLatestBlock - RollbackWindow, the most recentRblocks are still readable.Fis eventually deleted, each store reclaiming on its own schedule.Guarantee 2 is why pruning is to the shared minimum rather than to each store's
own boundary: a retained snapshot is only restorable if the blocks that follow it
survive in the contiguous stores. This is recorded on
StorageGarbageCollectorConfig.RollbackWindowand, in block terms, onBlockDBConfig.RetentionWindow.GetRetentionWindowreports extra retention beyond the sharedRollbackWindow, withInfiniteRetentionWindow(-1) meaning never prune.Note that
ReceiptStoreConfig.KeepRecent == 0already means "keep everything",which is the opposite of what
0means to the collector, so it is mapped toInfiniteRetentionWindowrather than passed through.StateWAL answers
0unconditionally, and has no retention config of its own. Itsdepth is not its to declare: it is a replay source, and SC/SS already express how
far back it must reach by answering their oldest live snapshot as a boundary. A
window here would be additive on top of the shared minimum, retaining every
managed store further back rather than the WAL alone — a fleet-wide decision
wearing a per-store name, which is what
RollbackWindowalready is.Snapshot stores
FlatKV restores only at a snapshot boundary and replays the WAL forward from
there, so what it must retain is not a block range but the newest snapshot at or
below the target.
GetPruningBoundaryreports that, which is what holds the WALback for it.
Config changes
littblock.BlockDBConfig.Retention→RetentionTime, default24h→1h.It is an age floor, not a retention policy; how much history BlockDB keeps is
RetentionWindow. TheAutobahnBlockDBConfig.Retentionoverride keeps itsname, since its
retentionJSON key is a persisted config format.littblock.DefaultConfignow leavesRetentionWindowat0(was10000).It is an input to a minimum shared across every managed store, so a non-zero
default here would have held ReceiptDB, the state WAL and the SC snapshots
10k blocks further back on BlockDB's say-so. Every other store reports 0; a
deployment wanting deeper block history sets it at the call site.
ExternalPruningonReceiptStoreConfigandFlatKVConfig, bothmapstructure:"-"and both defaulting tofalse.ReceiptStoreConfig.KeepRecentis deliberately left at0(keep everything).Nothing here couples it to
RetentionWindow, because the two fields disagreeabout
0: BlockDB folds only negatives toInfiniteRetentionWindow, so0there is the most aggressive setting, while ReceiptDB folds
<= 0to infinite,so
0there means never prune.KeepRecentcannot express "nothing beyond therollback window" at all. Reconciling the two is left to the wiring PR, along
with what the shared window should actually be.
Also in this PR
littblock.LittBlockConfigtoBlockDBConfig(ripples intosei-tendermint).ledgertoblocks. No migration is needed —BlockDB is not deployed on any network and no such data exists. Because the
table name is persisted layout rather than an identifier,
NewBlockDBwouldotherwise open a fresh empty table beside the old data, and an empty store is
indistinguishable from a correct one until something asks for history. A
refuseLegacyTablecheck at open turns that into a startup error naming thedirectory, for dev/CI/devnet homes written before the rename; it can be deleted
once no such directory remains.
seiwal.WAL.PruneBeforeas safe to call off the WAL owner'sgoroutine, unlike most of the interface, which is what lets the collector prune
the WAL from its own goroutine.
Not in this PR
Not moving all stores to use
StorageGarbageCollectoryet. We expect toconstruct it in a future PR when we decide to unify the pruning for mainnet. Both
ExternalPruningfields deliberately default tofalse: no behavior change bydefault.
That wiring PR is also where the remaining guard belongs. The receipt path can
reject its unsupported combination at startup, but nothing on the FlatKV path can
validate that a collector exists — "a collector exists" is not knowable from that
package. Keeping the field unreachable from config is what stands in for the
check until then.
Also worth knowing at wiring time: enabling
ExternalPruningchanges the shape ofsnapshot retention rather than just its depth. Snapshot count becomes roughly
RollbackWindow / SnapshotIntervalinstead ofSnapshotKeepRecent + 1.Not managing SS yet in this PR since SS doesn't have snapshot capability yet.
Testing
*_gc_test.gosuite per store, plus collector coverage for the self-pruningpath (a store reporting
falsekeeps its vote but receives noPruneBelow).litt_receipt_gcfilter_internal_test.gocovers the filter as a predicate andend-to-end: blocks below the floor are reclaimed by a real litt GC pass, those
at or above are retained. It fails if the filter is removed.
litt_receipt_pruner_internal_test.gopins when the local pruner runs, as apure predicate rather than a timing assertion.
litt_block_legacy_table_test.gocovers the pre-rename refusal, including thata refused open leaves the directory exactly as it found it.
testdata/*.golden, so each newvalue lands in a diff.
-race.