Skip to content

Preserve existing downloads when transfers fail - #199

Closed
jdotc1 wants to merge 3 commits into
mlmed:mainfrom
jdotc1:fix/torchxrayvision-staged-download-publication
Closed

jdotc1 wants to merge 3 commits into
mlmed:mainfrom
jdotc1:fix/torchxrayvision-staged-download-publication

Conversation

@jdotc1

@jdotc1 jdotc1 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

The download helper writes to the public cache filename before a transfer has succeeded. An interrupted first download can therefore leave a partial file that get_weights() treats as present on the next call: models.py checks os.path.isfile() before downloading, while the DenseNet and ResNet load-error handlers print the filename and re-raise without deleting it or retrying the download.

Stream into private sibling staging, check HTTP status and usable identity-encoded Content-Length, and replace the destination only after a successful transfer. Requests use connect/read inactivity timeouts. Responses and staging are closed on failures. This leaves an unsuccessful first download eligible for a later retry as well as preserving an existing destination during an explicit replacement attempt.

Successful replacement changes the destination inode and may change permission metadata; this does not authenticate model contents or provide crash durability.

Validation

Reused the retained eight-case focused handoff verification for the unchanged download implementation. Added tests/test_download.py to cover the actual get_weights() retry path: HTTP-status failure and an interrupted first stream each leave no cache file or staging directory, then a second call requests and publishes a complete response. Both subcases passed in an isolated source-function harness using the actual download() and get_weights() functions and the new repository test. Full package imports and the full test suite were not run locally; no model downloads or live endpoints were used.

@ieee8023

Copy link
Copy Markdown
Member

Thanks for the contribution. What is the failure mode that this addresses? It currently tries to download a file again if it failed to load it. So the file being preserved is no good. I'm not sure it is worth the extra complication.

@jdotc1

jdotc1 commented Oct 17, 2026

Copy link
Copy Markdown
Contributor Author

The concrete failure is an interrupted first download, rather than preserving an already-unloadable weight file. In the current call path, get_weights() downloads only when os.path.isfile(weights_filename_local) is false. DenseNet and ResNet catch a loading error, print the filename, and re-raise; neither removes the file nor retries the download. A partial file left by the first transfer can therefore prevent the next call from downloading again.

I added a regression test in 94529c2 for this exact path. Both HTTP-status failure and a stream interrupted after a partial chunk leave no cache file; a second get_weights() call then requests and publishes a complete response. The two subcases passed with the actual source functions in an isolated harness. The existing download implementation and its earlier verification are unchanged.

The staging directory's purpose is to keep an incomplete transfer from becoming the cache entry. It does not validate whether a successfully downloaded model is trustworthy or loadable.

@ieee8023

Copy link
Copy Markdown
Member

Ok, I'll investigate this more. Good catch!

@ieee8023

Copy link
Copy Markdown
Member

I added more commits on a new PR in #200. I'll close this PR and merge there.

@ieee8023 ieee8023 closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants