Adapt dega v2 - #423
Draft
cornhundred wants to merge 25 commits into
Draft
Adapt dega v2#423cornhundred wants to merge 25 commits into
cornhundred wants to merge 25 commits into
Conversation
Adds the viewer-independent core of the celldega_regular_grid_v1 profile: the tile geometry and the tile -> row-group -> file numbering. - x-major tile ids (tile_x * num_tiles_y + tile_y), matching Celldega's RowGroupTileReader, but reimplemented rather than imported so spatialdata-io gains no dependency on a viewer package (Celldega already depends on spatialdata-io, so importing it would be circular). A conformance test pins the two formulas together. - Half-open tile bounds, with the dataset's upper edge clamped into the last tile so points on x_max/y_max are not dropped. - Out-of-grid coordinates raise instead of being silently clamped, since they indicate a mismatched transform rather than a rounding artefact. - Zero-padded chunk filenames: Celldega indexes the manifest's files array by position while dask globs and sorts lexicographically, so unpadded names would make dask reorder partitions past 10 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FeatureCatalog assigns genes codes [0, n_genes) in the table's var_names order,
so a gene's feature_code is also its CBG row-group index -- no second lookup and
no browser-side string join. Non-gene features (controls, unassigned codewords)
are retained but coded above every gene and flagged, never folded into a real
gene. Verified on Xenium pancreas: 377 genes + 164 controls.
write_points_regular_grid keeps every canonical column and the index, and adds
display_xy (fixed_size_list<uint32>[2], already interleaved for deck.gl) and
feature_code. Rows are grouped by tile, one row group per logical tile including
empty ones, split across zero-padded chunk files.
Notable details:
- statistics are disabled: the tile formula is the spatial index, so no client
reads column-chunk min/max, and they inflate the footer the browser fetches.
- coordinates are cast to float64 before the affine: under numpy 2's NEP-50
promotion, float32 * python float stays float32 and loses pixel precision.
- the rewrite is idempotent. Re-running on an already-optimized element replaces
the render columns rather than appending duplicates, which previously produced
an unreadable file ("Multiple matches for FieldRef") because pandas also
round-trips fixed_size_list back as a variable-length list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds display_geometry (list<list<fixed_size_list<uint32>[2]>>: polygon -> rings -> interleaved integer pixel vertices) and cell_code, alongside the untouched canonical WKB geometry. - Cells are assigned to exactly one tile by centroid in display pixel space; a polygon whose outline crosses a tile boundary is not duplicated. Covered by a fixture case with exactly that shape. - display_geometry is marked lossy in the manifest: exterior ring only, largest part of a MultiPolygon, matching Celldega's current behaviour. - The nested layout is walked by a test that mirrors getPolygonDataFromChunk (polygon offset -> ring offset -> coordinate index), so a layout change that would break the browser fails here instead. - GeoParquet 'geo' metadata is preserved via the same conversion to_parquet uses. GeoDataFrame.to_arrow() emits GeoArrow extension metadata but not the 'geo' key, and without it the rewritten file stops being readable by geopandas.read_parquet and by SpatialData. Verified against Xenium pancreas: 140,702 cells, all single-ring Polygons, shapes index matches table obs_names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One row group per gene, so selecting a gene fetches a single row group instead of touching any transcript data. Schema matches Celldega's existing CBG reader exactly: cell_id / expression / gene, with gene_to_row_group and num_genes in the parquet schema metadata. Deliberate difference from Celldega's own writer: every catalog gene gets a row group in catalog order, including all-zero genes, so that 'row group index == feature_code' holds and one integer addresses both a transcript's gene and its expression vector. gene_to_row_group is still written for clients that look it up rather than assume it. Explicit stored zeros are dropped -- a sparse matrix can carry them and they are not expression. Validated against Xenium pancreas: 377 genes, 2,607,168 values, 16.2 MB, 0.2s; INS/GCG/ACTA2 cell sets and values match the AnnData table exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on Xenium pancreas (8.07M transcripts, 250px tiles), total overhead of the two render columns plus row-group fragmentation, against the 249.1 MB SpatialData default write: snappy 345.1 MB +38.5% 7.0s zstd 261.4 MB +4.9% 7.3s brotli 250.4 MB +0.5% 50.1s snappy barely compresses display_xy at all; zstd removes most of the overhead for no meaningful write cost, and brotli is 7x slower for another 4 points. Verified parquet-wasm 0.7.1 (celldega's pinned version) can read zstd: ZSTD = 5 is in its Compression enum and the zstd shims are compiled into the wasm bundle. This matters because an unsupported codec would fail only in the browser. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
add_spatial_tiling() tiles an existing store in place; xenium_spatially_tiled() goes from raw Xenium to a tiled store in one call. Both are single calls; the one-shot path is internally read -> write -> tile because the tiling rewrites written parquet. The manifest reuses Celldega's landscape_parameters.json keys (use_row_groups, tile_grid, row_group_files, technology, image_info) so its reader consumes it unchanged, and declares paths relative to the profile directory (../../points/transcripts/points.parquet) so Celldega can be pointed at that directory as base_url with no reader change. validate_manifest() fails loudly on the mistakes that would otherwise surface as a blank viewport: row-group counts disagreeing with the tile grid, file counts that cannot hold the declared row groups, CBG mappings pointing past the end, and declared files that do not exist. The re-runnability test caught the same idempotency bug in the shapes writer that was previously fixed for points: re-tiling appended duplicate display_geometry / cell_code columns rather than replacing them. Verified on Xenium pancreas: 15.0s to tile, 7,535 row groups across 19 files, read_zarr still returns 8,073,840 points / 140,702 shapes / (140702, 377) table, and canonical geometry is byte-identical after the reorder (including the 159 pre-existing invalid polygons, which are 10x source data, not corruption). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Normative description of celldega_regular_grid_v1: coordinate system, grid and tile numbering, row-group and multi-part file numbering, empty-tile handling, the render columns, feature and cell codes, CBG mapping, manifest layout, transport requirements, invalidation rules and a conformance checklist. Written viewer-independently with Celldega named as the reference client, and carrying the measurements behind the non-obvious choices (tile size, multi-file splitting, zstd) so the rationale survives the prototype. Also declares the 'columns' projection in the writer manifest fragments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… profile Writes a browser-ready WebP pyramid as Parquet row groups in Celldega's existing ImageRowGroupReader layout: zoom/tile_x/tile_y/image_data, one tile per row group, zoom_info in the schema metadata. Zoom numbering follows DeepZoom and was verified tile-for-tile against 'vips dzsave' on the same input: levels 0..11 for a 1300x700 image, level 11 = 3x2, level 10 = 2x1. So the output is interchangeable with tiles Celldega already produces. On Xenium pancreas it independently derives max_pyramid_zoom=16 and image_dimensions 34155x13770, matching the DegaFiles reference exactly. Encoding uses Pillow rather than libvips, so spatialdata-io gains no system dependency. Two memory measures matter on real images: the display window is taken from the smallest multiscale level rather than by scanning a 500-megapixel plane, and the 8-bit conversion runs in row blocks instead of materializing a float copy of the whole image. The canonical OME-Zarr image is untouched; this is explicitly a display cache, and the source element, dimensions, dtype, window, gamma, downsampling method and WebP settings are all recorded for invalidation. Also aligns the manifest with a real DegaFiles landscape_parameters.json (max_pyramid_zoom, image_dimensions, image_format, use_int_index, segmentation_approach, top-level tile_size). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ted memory Grouping rows by tile is a global sort -- a row at the end of the input can belong to the first tile -- so streaming the read alone is not enough. The streaming path makes two passes: stream the element one partition at a time and spill each row into a temporary file chosen by its destination chunk file, then sort each spill file independently and write its chunk. Peak memory scales with one partition plus one spill file rather than the dataset. Enabled automatically for partitioned (dask) elements; single-partition elements keep the single-pass path. A test asserts the two produce identical row groups, so they cannot silently diverge. Chunks pin the categorical dictionary to the element's full category list, since partitions observing different feature subsets would otherwise convert to incompatible Arrow dictionary types and could not be written to one file. Measured on Xenium Prime human skin (74,011,892 transcripts, 5,006 genes, 9.2x the pancreas): 94s to tile at 3.0 GB peak RSS, on a 17 GB machine where the in-memory path was estimated at 17-20 GB. 14,022 row groups across 36 files, all rows preserved, manifest validates, read_zarr returns the full element. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ixed-path assets Renames the profile from celldega_regular_grid_v1 to grid_files_v1. Naming it after one client contradicted the viewer-independent intent; nothing about the layout is Celldega-specific. Adds the files a client reads at fixed paths rather than through the manifest, which is what lets the profile directory stand in for a DegaFiles root so no client needs to know it is looking at a SpatialData store: - cell_metadata.parquet: per-cell centroids and names. Cells are the overview representation, so every centroid is needed up front. Centroids are not stored anywhere in a SpatialData store (the Xenium reader puts no x/y_centroid in obs), so they are derived from the shape geometry -- the same centroids already computed for tile assignment. Row order matches the table's obs order, so a cell's position here is its cell_code. - micron_to_image_transform.csv: the real micron-to-pixel affine. Coordinates are already in display pixels so this places nothing, but it drives the scale bar, and writing identity would make the scale bar wrong by the pixel size. - cell_clusters/: placeholder single-group clustering, since SpatialData does not require one and the Xenium reader does not load one. Without the file the viewer's category machinery fails on a 404. Also drops the design doc from the branch (it moves to the PR discussion) and removes supports_points_writer_hook, which was unused and untested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The render columns no longer go into the canonical elements. Canonical points and shapes are still re-ordered into tile row groups -- that is what makes spatial subsetting cheap from Python -- but keep only their own columns. display_xy and feature_code go to <profile>/trx, display_geometry and cell_code to <profile>/cell_seg, with the same row-group layout. This fixes two separate problems with one change: 1. SpatialData.write() round-trip. A nested Arrow column cannot survive dask's parquet round-trip: it either fails outright on a schema mismatch or silently comes back as a string. A tiled store could therefore not be rewritten. Now it can, and a test asserts it. 2. Column projection. parquet-wasm corrupts the IPC stream whenever 'columns' is passed (0.7.1 and 0.7.2, apache-arrow 15 and 18, scalar and nested alike, even an empty array), which is what left the viewer showing only the image layer. A standalone render file makes projection unnecessary: reading every column of it already transfers only what is drawn. Also in this change: - All image channels are written, not just the first. Xenium morphology_focus has four; each gets its own pyramid and image_info entry with a distinct colour. Channel names are sanitised for paths, since names like 'ATP1A1/CD45/E-Cadherin' would otherwise create nested directories. - Render files are written before the canonical rewrite. Rewriting canonical replaces the files the lazy dask frame still points at, so the other order made the second read fail. - The streaming path reported render_only=False regardless; both paths now report it, with a test covering each. - Re-tiling drops render columns left in canonical by the previous layout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e intensity Two fixes found by rendering the profile in a viewer. meta_gene.parquet had the wrong shape. A client reads it by convention, taking the gene list from the parquet *index* and colours from a 'color' column, alongside mean/std/max/non-zero. The profile wrote name/feature_code/is_gene as plain columns with no index and no colour, so the gene list came back empty: the viewer showed no transcript controls at all and logged nothing, because nothing had failed. It now matches that layout, with per-gene statistics computed column-wise from the sparse matrix (never densified -- a 5,006-gene panel would be several GB), and carries feature_code/is_gene alongside as profile additions. Image intensity is no longer stretched by default. The previous 1st-99.9th percentile window was applied on top of the viewer's own intensity slider and blew out the mid-tones: on a Xenium DAPI tile it took the mean from 1.3 to 37.4 with 1.2% of pixels saturated. The default is now the full dtype range, a linear mapping matching Celldega's pipeline, which saves raw values and leaves brightening to the slider. display_min/display_max still override it and are recorded in the manifest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pandas writes a named index as a column of that name. Our index was named 'name', so the parquet had a 'name' column and no '__index_level_0__' -- and a client looks for the gene list only under '__index_level_0__'. The gene list therefore still came back empty, with no error, exactly as before the schema fix. Leaving the index unnamed makes pandas emit '__index_level_0__', matching a DegaFiles meta_gene.parquet, whose index is also unnamed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… feature_code A client derives its integer gene id from a feature's row position in meta_gene.parquet, then colours transcripts by indexing an array built the same way. Sorting the frame by name interleaved the 164 control features among the 377 genes, so feature_code no longer matched that position and most colour lookups returned undefined -- which renders transparent rather than erroring. The catalog order is already the normative one (genes first, in var_names order, so a gene's feature_code is also its CBG row group), so writing the frame unsorted makes row position, feature_code and CBG row group all agree. Test uses a deliberately non-alphabetical var_names order with a control that sorts in the middle, so a reintroduced sort fails it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…integers Rounding to whole pixels put every transcript on an integer lattice. The displacement is scientifically negligible -- mean 0.08 um against Xenium's ~0.1-0.3 um localisation precision -- but visually obvious, because regular quantisation creates structure the eye reads as real. float32 was chosen over fixed-point integers despite storing less densely. On Xenium pancreas the render file measures: whole pixels (uint32) 27.6 MB 1 px max err 0.150 um fixed-point x4 (uint32) 43.3 MB 1/4 px max err 0.038 um float32 78.5 MB ~0.004 px max err 0 Fixed-point is denser, but requires the client to apply a scale read from the manifest; a client that ignores it renders everything silently offset by that factor. float32 needs no client-side arithmetic and cannot be misread. The extra ~50 MB is about 1% of a 3.5 GB store, which is a good trade for removing a whole class of silent-wrongness. Applies to display_geometry as well, so polygon vertices are no longer quantised either, and deck.gl still receives the Arrow buffer with no copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hook was prototyped in core and then dropped, since nothing used it: tiling operates on an already-written store. This comment was the last reference to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only the defaults are Xenium (element names, the technology string); the profile itself reads generic SpatialData elements, which is why it lives in spatialdata-io rather than beside the Xenium reader. Uses a store with different element names, a different micron-to-pixel scale, and a control feature absent from var_names, so a Xenium assumption creeping into the writer fails here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes what the viewer no longer needs written out, now that it reads obs, var, X and the OME-Zarr images directly: cbg_parquet.py -181 tables/table/X serves it webp_parquet.py -322 moved to celldega.pre; images read natively write_cell_metadata -57 obs + obsm["spatial"] serve it to_frame/_expression_stats -92 computed from X in the client cluster + transform writers obs categoricals and NGFF transforms serve them their tests -553 1352 deletions, 40 insertions. Also fixes a silent correctness bug the removal exposed. `feature_code` indexes the full catalog -- genes in `var` order, then controls -- but `var` holds only the genes. Xenium pancreas reaches code 539 against 377 genes, so 164 control probes indexed past the end of any colour table built from `var` alone and lost their colour without erroring. The manifest now records the names beyond `var`. The manifest also gains a `spatialdata` block declaring which components are read natively, which is what lets a client opt in per component. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reading one gene from a CSR matrix touches every chunk, so a client had to
download the whole matrix -- 4.5 MB for Xenium pancreas, 54.8 MB for Prime skin,
unbounded beyond that. Two additions fix it, both plain AnnData that round-trips:
var["mean"|"std"|"max"|"non_zero"] a gene list no longer needs X at all
layers["X_csc"] one gene is indptr[g]:indptr[g+1]
The chunking is the point. AnnData sizes chunks for whole-matrix reads -- 162,948
non-zeros per chunk against ~6,915 for one pancreas gene -- so a gene cost 24x
what it needed. write_csc_layer sizes chunks by the average gene instead.
Measured end to end, bytes fetched by the browser reader:
gene list 4.5 MB -> 0.02 MB
one gene 4.5 MB -> 0.048 MB
and the per-gene cost is now bounded by the gene, not by the matrix, which is
what makes it scale.
Two things worth knowing:
- SpatialData refuses to overwrite an element inside the store it was read from
(scverse/spatialdata#520), so the table is deleted and rewritten.
- the layer is written after the table and directly, so consolidated metadata
has to be refreshed or Python readers cannot see it. Browsers are unaffected,
since zarrita reads each node directly -- a difference that only shows up on
the Python side.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gene colours go to var["color"] and cluster palettes to uns["<column>_colors"]. A var column round-trips through AnnData and is visible to scanpy and anything else, where a viewer-specific colour file would be visible only to Celldega. The palette matches the fallback Celldega generates, so a store looks the same whether or not this ran. Worth flagging in review: a gene colour column is a new convention. AnnData has uns["<column>_colors"] for obs categoricals -- which add_cluster_colors follows exactly -- but nothing for genes. Also fixes a real bug: the README and design notes both document `from spatialdata_io.experimental import add_spatial_tiling`, which raised ImportError because experimental/__init__.py never exported it. Only the full module path worked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gene colours were going to a new var["color"] column. They now go to uns["gene_colors"], which is the shape AnnData already uses -- a <name>_colors list aligned to an ordering, exactly as scanpy writes for obs categoricals, just ordered by var_names instead of by category. This removes the only genuinely new convention from the proposal. Nothing here now asks a reader to learn a new idea. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ctory
Adds profile_layout="canonical". The v1 layout is unchanged; the new one writes
no display Parquets and no visualization/ directory, because the canonical data
can serve a viewer directly:
- shapes geometry is written as `geoarrow`, giving
list<list<struct<x, y>>> tagged geoarrow.polygon. That is the only encoding
@geoarrow/deck.gl-layers can read -- they accept the geoarrow extension types
and have no WKB support at all -- so the encoding is what decides whether a
browser can render canonical geometry without a display column.
- canonical points columns are ordered x, y, feature_name first. parquet-wasm
coalesces a projection into one byte range spanning the first to the last
requested column, so a column physically between them is fetched too:
measured 27.7 KiB with z in the middle against 19.7 KiB without.
- the manifest goes into the store's root Zarr attributes. Beyond removing the
directory, root attributes survive read_zarr(...).write(other_store) while a
sidecar directory does not, so the profile metadata now travels with the
store through an ordinary SpatialData round-trip.
Not attempted: moving the display columns into the canonical Parquets. A nested
Arrow column still does not survive dask's round-trip -- sdata.write() reports
success and returns display_xy as the string '[0.0, 0.0]' on re-read. Reading the
canonical columns directly avoids needing them at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #423 +/- ##
==========================================
+ Coverage 63.15% 69.14% +5.98%
==========================================
Files 26 34 +8
Lines 3257 4061 +804
==========================================
+ Hits 2057 2808 +751
- Misses 1200 1253 +53
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.