Repository navigation
registry: pull large blobs from mirrors that don't support byte ranges - #67
Merged
Merged
Conversation
Member
|
Merged, thanks @Outstanding2291. A small follow-up on main (38ae7bb) corrects the 408 test's comment: with a 40 MiB blob it retries the range request, whose 200 then falls back to a single stream. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some Hugging Face mirrors (Jfrog Artifactory's /resolve/ in particular) answer
a range request with the whole body as 200 instead of 206 Partial
Content. The puller treated any non-206 as fatal "corrupt data", so a
model whose weights blob exceeds the single-stream threshold (32 MiB)
could never be pulled from such a mirror — which is the default for any
large model hosted there.
Two changes:
of failing. Ranges are only a download strategy; the blob is streamed
once and verified by digest as before. A new Error::NoRanges signal
replaces the misleading Corrupt error so the fallback is explicit.
backoff (it was treated as terminal, since is_retryable only retried
connect/body/timeout/5xx). A mirror or proxy that abandons the
transfer transiently no longer kills the pull outright.
Covers both the range path (fetch_ranges) and the single-stream path
(fetch_whole): both gate retries on Error::is_retryable.
Regression tests: pulling a blob from a server that ignores Range, and
one that answers a single 408 before serving the blob.