Add Parquet and DuckDB output formats

[amc-corey-cox]

Closes #325.

Transformed objects stream to a staging JSONL file, which DuckDB's read_json_auto then turns into the artifact. That inference is the point — it types each field on its own, so a nested object lands as a STRUCT column and a list of them as STRUCT[]:

value_quantity => STRUCT(value_decimal DOUBLE, unit VARCHAR)
observations   => STRUCT(id VARCHAR, "value" DOUBLE)[]

A consumer reads a nested field with a column access rather than a join, and linkml-map never has to describe the shape. Staging preserves the streaming contract: objects are never all held in memory, and the output path only ever contains the finished artifact.

linkml-map map-data -T transform.yaml -s schema.yaml --source-type Person \
  -o agents.parquet data/Person.tsv

Both formats work by extension and as -O targets alongside a text primary output. DuckDB output writes one table per run — named by --table-name, defaulting to the file stem — so repeated runs against one path build a database covering several target classes, which is the shape downstream consumers assemble by hand today.

duckdb was already a hard dependency (the join engine uses it), so this adds none.

Why this rather than the SQL backend

#325 framed Parquet as work on DuckDBTransformer. It isn't: SQLCompiler supports almost none of the specification language — expr, value, case(), unit conversion and value mappings are all dropped silently — so nothing real transforms through it. The Python backend already produces correct nested output; only the serialization step was missing. See the closing comment on #332 for the full finding.

Incidental refactor

StreamWriter gains two optional hooks, staging_path and finalize_path. The tabular header rewrite moves onto finalize_path, so MultiStreamWriter no longer type-checks for TabularStreamWriter to decide whether to post-process.

[Copilot]

Pull request overview

This PR adds first-class columnar outputs (Parquet and DuckDB) to linkml-map’s streaming writer pipeline by staging transformed objects as JSONL and letting DuckDB infer nested types (STRUCT / STRUCT[]), enabling nested-field access without joins and without holding all rows in memory.

Changes:

• Introduces ParquetStreamWriter and DuckDBStreamWriter built on a new ColumnarStreamWriter staging + finalize flow.
• Refactors StreamWriter/MultiStreamWriter to support per-writer staging paths and finalize_path() post-processing (including moving tabular header rewrites there).
• Adds tests covering nesting preservation, staging cleanup, DuckDB table naming, and multi-output fan-out.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
tests/test_writers/test_columnar_output.py	Adds coverage for Parquet/DuckDB outputs, nesting types, staging cleanup, and multi-run/table behavior.
src/linkml_map/writers/output_streams.py	Adds new output formats and columnar writers; introduces staging/finalize hooks and refactors post-processing.
src/linkml_map/cli/cli.py	Threads --table-name through streaming CLI path and uses updated writer creation for primary output.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/linkml_map/writers/output_streams.py:642

• ColumnarStreamWriter.finalize_path() only unlinks the staging JSONL on success. If DuckDB conversion raises, the .staging.jsonl file is left behind (potentially large/sensitive) and the docstring’s “then remove the staging file” guarantee no longer holds. Consider cleaning up the staging file in a finally block (logging a warning if removal fails) so failures don’t strand temp artifacts.

        try:
            connection.execute(self._conversion_sql(target, staged))
        finally:
            connection.close()
        staged.unlink()

src/linkml_map/writers/output_streams.py:639

• DuckDB conversion here uses a bare duckdb.connect(...), but other parts of the codebase intentionally use linkml_map.utils.lookup_index.make_connection() to apply container/cgroup-aware settings (see src/linkml_map/transformer/engine.py:119 and src/linkml_map/utils/lookup_index.py:160-166). For large JSON→Parquet/DB conversions, skipping those settings can increase the risk of OOM or unpredictable resource usage in constrained environments. Consider reusing the same connection configuration for these conversions (including the file-backed case).

This issue also appears on line 638 of the same file.

        import duckdb

        connection = duckdb.connect(*self._connect_args(target))
        try:
            connection.execute(self._conversion_sql(target, staged))

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.