Repository navigation
Disk space reduction - #56
Merged
Merged
Conversation
Add cx.compress_h5ad(src, dst), a lossless gzip+shuffle re-encode of any .h5ad. It walks the HDF5 tree directly (independent of AnnData), compresses chunks on a thread pool through write_direct_chunk, verifies every byte and attribute through h5py before replacing dst, and keeps memory bounded. The result opens in any HDF5 reader without plugins. Stream compressed inputs faster: iter_matrix_chunks inflates deflate (with or without shuffle) chunks on a thread pool for fast-axis reads of backed CSR/CSC and row reads of dense matrices. Blocks are identical to plain slicing; uncompressed files keep the old path. Stop writing duplicate data. Every stored value stays bit-identical: - nb_glm_test: drop logfoldchanges (== X) and, without shrinkage, logfoldchange_raw; drop *_ln layers when lfc_base="ln"; store a global dispersion once in var; converged as bool, iterations as a small int. - pts_rest is a var column in every DE output. - shrink_lfc and batch_process no longer duplicate X as a layer. - t_test no longer writes uns matrices that its final write discarded; streaming Wilcoxon no longer builds a dense placeholder X. Make resume=True resume interrupted DE runs in every method and path. Partial results used to live in a temporary directory an interrupted run took with it, so nb_glm_test returned NaN rows for checkpointed perturbations, t_test zeros, and wilcoxon_test refused to resume. ResumableRun keeps them beside the output with a fingerprinted checkpoint; streaming Wilcoxon builds its file there and moves it into place only when complete. Also fix: - tl.rank_genes_groups(method="t-test") returns t_test's own result instead of a rebuilt one with the wrong logfoldchanges, zero pts and pts_rest, and a bloated recarray rewrite; t_test gains corr_method. - A reloaded t_test result keeps its effect size. - Wilcoxon uns metadata is anndata-encoded, so adata.uns and the plotting readers see it. - The QC gene-filter cache and sort_by_perturbation's .meta.h5ad no longer leak when a run fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Re-execute the tutorials on 0.1.6. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Write every Wilcoxon result under a partial name and rename it into place, so a killed write never leaves a file that loads as finished. - Fingerprint array and pandas arguments by content digest instead of a truncated repr or a full tolist(). - Leave memory and chunking knobs out of the fingerprint, so a run killed for memory resumes with a lower limit. - t_test: fill untested rows with NaN (was p = 0), copy results out of the memmaps before deleting them, validate corr_method up front. - Name the resume directory ".<output>.resume" so it no longer collides with the converters' ".<output>.partial" file. - compress_h5ad: copy and verify committed (named) datatypes. - Resolve n_jobs through joblib in one helper, and share one decode thread pool across streaming reads. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Save checkpoints with os.replace so repeated saves work on Windows. - Fingerprint nb_glm runs with the resolved freeze_control and streaming control, so a resume never mixes models. - Write t_test and nb_glm_test outputs under a partial name and rename on success; unmap nb_glm memmaps before discarding the resume directory. - Checkpoint t_test once progress crosses the interval, not on exact multiples. - Restart streaming Wilcoxon when its work file is unreadable, and write the control pts_rest once. - Size the decode pool lazily and cap it by OMP_NUM_THREADS. - Drop the duplicate layer list, shorten the resume docstrings and set the 0.1.6 release date in the changelog. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, comment @claude review on this pull request, or push a new commit, to trigger a review.
The Python 3.12 CI job resolves anndata 0.13 and pandas 3, which change two things crispyx and its tests relied on: - Backed AnnData exposes X as layers[None], so the shrink_lfc disk estimate counted X twice and the multi-channel batch test saw an extra None layer key. Count and compare only named layers. - pandas 3 string columns are written as nullable-string-array groups (values + mask), so the DE resume comparison helper now walks every dataset under a group instead of reading the group as an array. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.