feat: pertubation predictive score - #284
Open
ori-kron-wis wants to merge 21 commits into
Open
ori-kron-wis wants to merge 21 commits into
ori-kron-wis wants to merge 21 commits into
Conversation
Implement the abstract base class and naive mean-delta baseline predictor for perturbation-response evaluation. MeanBaseline predicts the mean training-perturbation delta for all requested perturbations, serving as the evaluation floor baseline from the perturbation literature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Added two perturbation evaluation metrics: - systema_decomposition: scores shared vs. specific effect components separately - combination_additivity: wraps pertpy's evaluate_combinations for additivity scoring Handle edge cases where constant arrays produce undefined correlations. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds PerturbationBaselines/PerturbationMetrics config dataclasses and the PerturbationBenchmarker orchestrator that fits mean/additive/linear baselines plus user predictions and scores them through the shared metric surface. Fixes a real bug found while running the brief's own test fixture: _compute_true_deltas control-diffed adata_test alone, but the documented adata_test shape (held-out perturbation cells only, no control cells) makes pertpy's compute_control_diff raise `Reference key control not found`. Control cells are now borrowed from adata_train when absent from adata_test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…urbationBenchmarker.benchmark() Raise ValueError if predictions contain a key matching an enabled baseline name (mean/additive/linear) to prevent silent mislabeling of user models as baselines and loss of baseline scores. Added test case verifying collision raises ValueError. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add public get_ground_truth_significance() accessor to retrieve ground-truth significance diagnostic - Add plot_results_table() method to visualize benchmarking results as a styled table - Include necessary imports for matplotlib, plottable, and os modules - All 10 tests pass (4 new tests for the new methods) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The test_public_api_is_importable() was checking 10 of 11 symbols in __all__, missing BasePerturbationPredictor (the ABC from Task 3). Add it to both the import statement and assertion tuple to verify all 11 exports. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
pertpy.tools._coda._sccoda unconditionally calls
jax.config.update("jax_enable_x64", True) at import time, regardless of
whether Sccoda is used. Importing pertpy anywhere in a pytest session (or
any process) silently flips JAX's global float precision, breaking
scib_metrics.utils._kmeans/_lisi's jax.lax.while_loop carry-dtype
assumptions in unrelated, already-collected test modules.
Fixed at two layers: import_pertpy() now restores the prior x64 setting
after importing pertpy (protects real library usage), and a new
tests/perturbation/conftest.py neutralizes the same mutation before pytest
collects any test in that directory, since test files import pertpy
directly via pytest.importorskip (bypassing import_pertpy()) at collection
time. Reproduced and fixed locally: 42/42 passed across the perturbation
suite plus tests/test_metrics.py::test_kmeans,
test_nmi_ari_cluster_labels_kmeans, and the full tests/test_benchmarker.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… scaling defaults, DE-index leakage) - combination_additivity: pseudobulk train+test union (finds held-out combos), add get_combination_additivity() accessor, forward dict kwargs (sep, etc.) - ground truth significance empty-fallback now matches pertpy's real 5-column schema - get_results/plot_results_table default to min_max_scale=False, matching Benchmarker - true_de_gene_indices no longer silently restricts delta_correlation unless de_rank_recovery is also enabled - warn once when systema_decomposition is enabled but <2 held-out perturbations exist Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #284 +/- ##
==========================================
+ Coverage 91.41% 92.33% +0.91%
==========================================
Files 27 32 +5
Lines 1095 1409 +314
==========================================
+ Hits 1001 1301 +300
- Misses 94 108 +14
🚀 New features to boost your workflow:
|
…hmarker Mirrors tests/test_benchmarker.py::test_benchmarker's pattern: run the full default pipeline (all baselines, all default-on metrics) end-to-end on synthetic data, assert get_results(), and exercise plot_results_table() the same way. Uses a train/test split holding out two perturbations so systema_decomposition actually runs instead of skipping, and a combination present in the train+test union so combination_additivity finds a real result. Also fixes test_plot_results_table_returns_a_table back to show=False (matplotlib GUI backends are unreliable inside pytest regardless of this flag; visually inspecting the plot requires running it outside pytest) and removes a stray table.plot_results_table(...) call left over from manual edits (Table has no such method). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
show=True opens a real, blocking GUI window (confirmed identical to Benchmarker.plot_results_table's behavior via direct comparison -- both block ~10s waiting for a window close with the macosx backend run from a plain interpreter). That's unreliable to depend on inside any test runner. save_dir writes an inspectable SVG regardless of backend/runner quirks; rendered and visually verified the resulting table (baseline-row labeling, PRGn colormap circles) matches the intended design. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.