Parquet output from the ingestion tooling (tracks linkml-map#325)

[amc-corey-cox]

The metadata index is published as Parquet on object storage (see ARCHITECTURE.md, #12), but the tooling that produces it emits DuckDB native database files. Parquet output does not exist yet.

Tracking here because it blocks our storage design, even though the work happens upstream:

• Emit Parquet as an output format, not only DuckDB native files linkml/linkml-map#325 — emit Parquet as an output format
• Flesh out DuckDBTransformer as a full execution backend linkml/linkml-map#151 — DuckDBTransformer as a full execution backend, the surrounding work

The reason we need Parquet rather than native files is version coupling. DuckDB's native storage format is backward compatible but only best-effort forward compatible, and our producer (Python ingestion tooling) and consumer (service tier) upgrade independently. A producer running a newer DuckDB can write an artifact the consumer cannot open, and it fails at the consumer's deploy rather than at build. Parquet has no such coupling, and it keeps the index readable by the wider ecosystem rather than by one engine.

Until this lands, the Parquet-on-S3 read pattern in #14 has to be exercised against files produced some other way — hand-converted, or written directly with DuckDB's COPY ... TO ... (FORMAT PARQUET). That is fine for the spike and shouldn't block it.

[amc-corey-cox]

Upstream dependency is satisfied: linkml/linkml-map#333 adds Parquet and DuckDB as output formats to map-data, so the ingestion tooling gets both directly.

linkml-map map-data -T transform.yaml -s schema.yaml --source-type Person \
  -o agents.parquet data/Person.tsv

Nesting survives as typed columns — value_quantity comes through as STRUCT(value_decimal DOUBLE, unit VARCHAR), and a multivalued nested slot as STRUCT(...)[] — so the metadata index design's requirement that a Quantity lookup be a column access rather than a join holds.

DuckDB output writes one table per run, named by --table-name or the file stem, and repeated runs against the same path accumulate tables. That means api/load_seed_data.py can be replaced by one map-data invocation per class, since the mechanism it uses (read_json_auto over the harmonized JSONL) is now what the writer does internally.

Note linkml/linkml-map#325 was closed against #333, and linkml/linkml-map#151 — the DuckDB execution backend — was closed as won't-do; it was never on the path to this. Worth waiting for #333 to merge before rewiring the loader.

[amc-corey-cox]

Both upstream blockers are done: linkml/linkml-map#325 (Parquet output) closed as completed on 2026-08-25, and linkml/linkml-map#151 (DuckDBTransformer backend) is closed. The ingestion tooling emits Parquet now, so there's nothing left to track here.

Reopen as implementation work if adopting Parquet in the metadata index turns up anything beyond consuming what the tooling already produces.