fix(download): adopt unmarked caches for pinned revisions - #977
Alex-Wengg merged 3 commits into
Conversation
I validate every cached compiled bundle and root file against the pinned listing before adopting a legacy cache. I remove incomplete bundles and stale sidecars beside kept files, and tolerate missing-file races. I preserve existing main and marked-cache behavior and cover both download entry points with regression tests.
|
Thanks! Two things to fix before merge: the repo-root fallback in |
I verify legacy files against Hugging Face LFS SHA-256 or Git blob SHA-1 identities using bounded reads, including empty and unknown-size files. I check nested plain files individually and remove incomplete compiled bundles at any depth before writing the revision marker. I limit repo-root fallback to flattened repo downloads and filter the validated list back to the requested paths so other variants are fetched only when their own next load needs them. Validation: RED had 19 assertion failures across 25 tests on the parent. GREEN passes 38 focused tests, 74 selected cache/download/progress tests, and 25 downloader tests. Full swift build and strict format lint pass. I also confirmed lfs.oid and oid in the live pinned Hugging Face tree.
I remove partial and etag sidecars for rejected or missing legacy files before writing the pinned revision marker. This prevents finished stale partials from restoring bytes that failed content verification. I cover rejected and missing destinations, and a repo download that refetches its requested JSON while leaving another bad variant absent. Validation: RED compiled and produced 9 assertion failures across 30 tests on c6bdeb3. GREEN passes 40 focused, 76 selected, and 25 downloader tests. Full swift build and strict format lint pass.
|
Thanks for the careful review. I pushed two commits on top (c6bdeb3 and 617eaa1) that cover all five points:
617eaa1 also removes the New tests cover each case. |
Fixes #976.
Caches written before revision markers existed are now reused when their files match the pinned listing, instead of being deleted and downloaded again.
When a pinned repo's directory has no
.fluidaudio-revisionmarker,ModelHublists every bundle and root file already on disk (not only the ones the current variant asked for) and compares each file's size with the size in the tree listing. A bundle with any missing or mismatched file is removed as a whole, soallModelsExiststays false and the nextloadModelsstill downloads it, even if the process dies before the refetch. Files that match are kept, stale.partialand.etagfiles next to them are removed, and the marker is written. The normal download loop then fetches only what is missing.The full-tree check matters for
speaker-diarization, where the streaming and offline models share one directory and one marker. Without it, a streaming load would mark the directory current and the offline bundles would never be compared.Unchanged:
mainrepos, caches whose marker matches, caches whose marker names a different revision (still wiped), offline mode, and the purge and force-redownload paths. No public API changes, andFileDownloaderis untouched. Comparison is by byte size, not hash. A removed bundle loses its partial downloads, since it is being discarded anyway.The adoption loop treats a file that disappears between the existence check and the remove (another load adopting the same directory) as already removed, so the race no longer surfaces as an error that triggers
loadModels' purge.Tests:
ModelCacheLegacyAdoptionTestscovers both download entry points, a stale bundle from a sibling variant, a bundle with a missing or wrong inner file (checked at theallModelsExistgate thatloadModelsOnceuses), sidecar cleanup,mainand existing markers, and concurrent adopters.TreeStubURLProtocolnow serves pinned revisions as well asmain. TheModelCache,ModelHub, subdirectory, progress and downloader suites pass (90 tests), along withswift buildandswift format lint --strict.