0.1.7: reuse a finished result only when its inputs match

CITATION.cff

@@ -4,8 +4,8 @@ title: "crispyx: Streaming analysis of large-scale CRISPR and Perturb-seq screen
 authors:
   - family-names: Du
     given-names: Jin-Hong
-version: 0.1.6
-date-released: 2026-09-26
+version: 0.1.7
+date-released: 2026-10-03
 url: "https://github.com/jinhongdu-lab/crispyx"
 repository-code: "https://github.com/jinhongdu-lab/crispyx"
 license: MIT

docs/api.rst

@@ -106,10 +106,9 @@ Generic batch statistics
 Differential expression
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-All three DE functions accept a ``force: bool = False`` parameter (v0.0.3+).
-When the expected output ``.h5ad`` file already exists on disk, the function
-reloads and returns the saved result instead of rerunning.  Pass ``force=True``
-to overwrite.  See :ref:`auto-reload` in the usage guide.
+All three DE functions reload an existing output ``.h5ad`` instead of
+rerunning when it was computed from the same input and arguments; pass
+``force=True`` to recompute.  See :ref:`auto-reload` in the usage guide.
 
 .. automodule:: crispyx.de
    :members:

docs/changelog.rst

@@ -1,6 +1,23 @@
 Changelog
 =========
 
+Version 0.1.7
+-------------
+
+*Released 2026-10-03.*
+
+* **Fixed: stale results.** The DE functions, ``batch_process``, the
+  pseudo-bulk functions, ``sort_by_perturbation`` and ``standardize_dataset``
+  could return an existing output after its input or a result-shaping
+  argument (e.g. ``perturbations``) had changed. Each output now records
+  the input file, every result-shaping argument, the crispyx version and its
+  layout in ``uns["crispyx"]``, and is reused only when these match; the
+  notice says why one is recomputed. Execution-only arguments
+  (``chunk_size``, ``memory_limit_gb``, ``n_jobs``, ...) do not invalidate a
+  result.
+* Outputs from 0.1.6 or earlier carry no such record and are recomputed once
+  after upgrading.
+
 Version 0.1.6
 -------------
 

docs/faq.rst

@@ -285,18 +285,32 @@ HPC / SLURM tips
 My DE result is loaded instantly on the second call — is that expected?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Yes.  Since v0.0.3 all three DE functions auto-reload an existing result file
-instead of rerunning the analysis.  When ``verbose=True`` a notice is printed:
+Yes, when nothing that shapes the result has changed.  The DE functions,
+``batch_process`` and the pseudo-bulk functions reuse an existing output file
+instead of recomputing it.  When ``verbose=True`` a notice is printed:
 
 .. code-block:: text
 
-   [crispyx] Loading existing result: data/crispyx_wilcoxon.h5ad
-   [crispyx] Pass force=True to rerun the analysis.
+   [cx] Loading existing result: data/crispyx_wilcoxon.h5ad
+   [cx] Pass force=True to rerun the analysis.
 
-If you changed a parameter (e.g. ``min_pct_ctrl``, ``min_pct_pert``, a covariate list, or
-``dispersion_scope``) and want the result to reflect the new settings, pass
-``force=True`` to the DE function.  The existing output file will be
-overwritten.
+Since v0.1.7 every output records what it was computed from in
+``uns["crispyx"]``: the input file (path, size and modification time), every
+argument that shapes the result, the crispyx version that wrote it, and the
+file layout.  The file is reused only when all of these match the call, so
+relabelling the cells of the input in place, changing ``min_pct_ctrl`` or
+``perturbations``, or upgrading to a crispyx that stores results differently
+recomputes it, and the notice says why:
+
+.. code-block:: text
+
+   [cx] Existing result at data/crispyx_wilcoxon.h5ad is not reused: the input file has changed since; rerunning.
+
+Arguments that only change how a run is carried out (``chunk_size``,
+``memory_limit_gb``, ``n_jobs``, ``verbose``, ...) do not invalidate a
+result.  Pass ``force=True`` to recompute regardless, for example after
+changing a ``batch_process`` reducer without changing its
+``statistic_name``.
 
 Can I pickle / serialise a ``RankGenesGroupsResult``?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/notebooks/batch_statistics_tutorial.ipynb

@@ -16,10 +16,10 @@
    "id": "setup",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:09.613903Z",
-     "iopub.status.busy": "2026-09-26T04:30:09.613829Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.077778Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.077491Z"
+     "iopub.execute_input": "2026-10-03T14:05:50.433660Z",
+     "iopub.status.busy": "2026-10-03T14:05:50.433424Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.902938Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.902629Z"
     }
    },
    "outputs": [],
@@ -59,10 +59,10 @@
    "id": "create-data",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.079640Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.079480Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.089275Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.089019Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.904551Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.904425Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.913041Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.912751Z"
     }
    },
    "outputs": [],
@@ -134,10 +134,10 @@
    "id": "std-reducer",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.090726Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.090646Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.093606Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.093394Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.914338Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.914252Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.917206Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.916996Z"
     }
    },
    "outputs": [],
@@ -180,10 +180,10 @@
    "id": "run",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.094892Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.094806Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.130341Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.130085Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.918409Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.918335Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.950676Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.950395Z"
     }
    },
    "outputs": [
@@ -192,8 +192,8 @@
      "output_type": "stream",
      "text": [
       "[cx] tl.batch_process: 3 groups × 6 genes, stratified by 'batch'\n",
-      "[cx] tl.batch_process: estimated disk usage: 317 B (3.7 TB free)\n",
-      "[cx] tl.batch_process: format_mismatch_policy='auto': streaming from the source (2 chunks x 16.8 KB at 371.3 MB/s = 0.00 s per re-read, 0.0 s of re-reading beyond the first chunk)\n"
+      "[cx] tl.batch_process: estimated disk usage: 317 B (3.9 TB free)\n",
+      "[cx] tl.batch_process: format_mismatch_policy='auto': streaming from the source (2 chunks x 16.8 KB at 398.8 MB/s = 0.00 s per re-read, 0.0 s of re-reading beyond the first chunk)\n"
      ]
     },
     {
@@ -209,7 +209,7 @@
      "output_type": "stream",
      "text": [
       "\r",
-      "tl.batch_process: 100%|██████████| 2/2 [00:00<00:00, 339.04gene chunk/s]"
+      "tl.batch_process: 100%|██████████| 2/2 [00:00<00:00, 356.51gene chunk/s]"
      ]
     },
     {
@@ -345,10 +345,10 @@
    "id": "verify",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.131775Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.131675Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.136069Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.135834Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.952132Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.952039Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.956262Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.956069Z"
     }
    },
    "outputs": [
@@ -390,10 +390,10 @@
    "id": "021badc9",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.137420Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.137341Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.159871Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.159635Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.957541Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.957422Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.981256Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.981054Z"
     }
    },
    "outputs": [
@@ -402,8 +402,8 @@
      "output_type": "stream",
      "text": [
       "[cx] tl.batch_process: 2 groups × 6 genes, stratified by 'batch'\n",
-      "[cx] tl.batch_process: estimated disk usage: 211 B (3.7 TB free)\n",
-      "[cx] tl.batch_process: format_mismatch_policy='auto': streaming from the source (2 chunks x 16.8 KB at 371.3 MB/s = 0.00 s per re-read, 0.0 s of re-reading beyond the first chunk)\n"
+      "[cx] tl.batch_process: estimated disk usage: 211 B (3.9 TB free)\n",
+      "[cx] tl.batch_process: format_mismatch_policy='auto': streaming from the source (2 chunks x 16.8 KB at 398.8 MB/s = 0.00 s per re-read, 0.0 s of re-reading beyond the first chunk)\n"
      ]
     },
     {
@@ -419,7 +419,7 @@
      "output_type": "stream",
      "text": [
       "\r",
-      "tl.batch_process: 100%|██████████| 2/2 [00:00<00:00, 551.74gene chunk/s]"
+      "tl.batch_process: 100%|██████████| 2/2 [00:00<00:00, 543.76gene chunk/s]"
      ]
     },
     {
@@ -560,10 +560,10 @@
    "id": "5175156a",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.161196Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.161100Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.165277Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.165060Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.982640Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.982539Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.986564Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.986363Z"
     }
    },
    "outputs": [
@@ -607,10 +607,10 @@
    "id": "b12b08cd",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.166588Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.166498Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.168891Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.168685Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.987861Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.987766Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.990087Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.989889Z"
     }
    },
    "outputs": [
@@ -669,10 +669,10 @@
    "id": "0bedccfb",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.170143Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.170066Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.176966Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.176729Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.991293Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.991223Z",
+     "iopub.status.idle": "2026-10-03T14:05:51.998176Z",
+     "shell.execute_reply": "2026-10-03T14:05:51.997906Z"
     }
    },
    "outputs": [],
@@ -707,10 +707,10 @@
    "id": "80f3891b",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.178266Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.178185Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.200739Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.200522Z"
+     "iopub.execute_input": "2026-10-03T14:05:51.999400Z",
+     "iopub.status.busy": "2026-10-03T14:05:51.999326Z",
+     "iopub.status.idle": "2026-10-03T14:05:52.022380Z",
+     "shell.execute_reply": "2026-10-03T14:05:52.022186Z"
     }
    },
    "outputs": [
@@ -719,8 +719,8 @@
      "output_type": "stream",
      "text": [
       "[cx] tl.batch_process: 2 groups × 6 genes, stratified by 'lane'\n",
-      "[cx] tl.batch_process: estimated disk usage: 422 B (3.7 TB free)\n",
-      "[cx] tl.batch_process: format_mismatch_policy='auto': streaming from the source (2 chunks x 9.7 KB at 311.7 MB/s = 0.00 s per re-read, 0.0 s of re-reading beyond the first chunk)\n"
+      "[cx] tl.batch_process: estimated disk usage: 422 B (3.9 TB free)\n",
+      "[cx] tl.batch_process: format_mismatch_policy='auto': streaming from the source (2 chunks x 9.7 KB at 322.0 MB/s = 0.00 s per re-read, 0.0 s of re-reading beyond the first chunk)\n"
      ]
     },
     {
@@ -736,7 +736,7 @@
      "output_type": "stream",
      "text": [
       "\r",
-      "tl.batch_process: 100%|██████████| 2/2 [00:00<00:00, 737.98gene chunk/s]"
+      "tl.batch_process: 100%|██████████| 2/2 [00:00<00:00, 726.48gene chunk/s]"
      ]
     },
     {
@@ -889,10 +889,10 @@
    "id": "1f6e4840",
    "metadata": {
     "execution": {
-     "iopub.execute_input": "2026-09-26T04:30:11.201996Z",
-     "iopub.status.busy": "2026-09-26T04:30:11.201923Z",
-     "iopub.status.idle": "2026-09-26T04:30:11.261170Z",
-     "shell.execute_reply": "2026-09-26T04:30:11.260930Z"
+     "iopub.execute_input": "2026-10-03T14:05:52.023540Z",
+     "iopub.status.busy": "2026-10-03T14:05:52.023464Z",
+     "iopub.status.idle": "2026-10-03T14:05:52.085261Z",
+     "shell.execute_reply": "2026-10-03T14:05:52.085011Z"
     }
    },
    "outputs": [