undatum

A command-line tool for data processing and analysis

Version: 1.7.0

undatum (pronounced un-da-tum) is a CLI for converting, analyzing, validating, and transforming datasets across many formats, with a streaming-first design for large files.

Features

• 140+ formats via iterabledata: CSV, JSON, JSON Lines, BSON, XML, XLS/XLSX, Parquet, AVRO, ORC, plus geospatial, lakehouse (Delta/Iceberg/Lance/DuckLake/Paimon), scientific, RDF, log, config, graph, and feed formats. Run undatum formats list to see every supported format and its read/write capabilities.
• Compression support: GZ, XZ, BZ2, ZIP, ZSTD, LZ4, 7Z, Brotli, Snappy, LZO
• Multi-cloud I/O: Read and write s3://, gs:///gcs://, and az:///abfs:///abfss:// URIs natively via iterabledata (pip install "undatum[cloud]")
• Database sources: Read from PostgreSQL, MySQL/MariaDB, SQLite, MS SQL Server, ClickHouse, MongoDB, and Elasticsearch/OpenSearch (undatum db query, undatum db dump)
• Optional TUI and web UI: Explore a bounded sample in the terminal (undatum tui) or a local browser (undatum web)
• Low memory footprint: Streams data for efficient processing of large files
• Automatic detection: Encoding, delimiters (comma, semicolon, tab, pipe), and file types
• Frictionless Data Packaging: Create, extend, and validate datapackage.json descriptors (undatum package)
• Data validation: Built-in rules for emails, URLs, and custom validators
• Ad-hoc SQL on files: Run DuckDB SQL over CSV, JSONL, Parquet, and other formats (undatum sql)
• AI-powered tooling: ai doc / ai filter via iterabledata (OpenAI, Anthropic, Gemini, Azure, OpenRouter, Ollama, LM Studio, Perplexity). Legacy --autodoc on analyze / schema / doc supports openai, openrouter, ollama, lmstudio, and perplexity.
• Agent tools & MCP server: JSON tools, LangChain StructuredTools, or undatum mcp serve
• Optional Data API: Serve file-backed datasets over HTTP (FastAPI + DuckDB)

Documentation

The full documentation site (Docusaurus) lives in docs/ and is published at datenoio.github.io/undatum.

Section	What it covers
Getting started	Install, quick start, positioning
Cookbook	Task index by role
CLI reference	Every command
Formats	Honest capability matrix
Python SDK	Fluent Dataset API
MCP / agents	Agent tools and MCP server
Troubleshooting	Exit codes and common errors

Source pages: docs/docs/. Changelog: CHANGELOG.md. Contributor workflow: WORKFLOW_GUIDE.md.

Installation

Using uv or pipx (recommended for CLI use)

uv tool install undatum
# or
pipx install undatum

Using pip

pip install --upgrade pip setuptools
pip install undatum

Dependencies are declared in pyproject.toml and will be installed automatically by modern versions of pip (23+), including pyarrow for Parquet.

macOS

brew install pipx && pipx install undatum
# or
uv tool install undatum

Release tags publish PyInstaller single-file binaries (Linux, macOS, Windows) on GitHub Releases. pipx/uv remain the supported install paths for most users.

A man page ships with the package (man undatum after install, or make man to regenerate man/undatum.1).

Optional extras

Extra	Enables
api	Data API server (undatum api, FastAPI + uvicorn + httpx)
extract	Document extraction (undatum extract, PDF/DOC/DOCX tables and text)
plot	Plotting (undatum plot, matplotlib)
tui	Interactive terminal UI (undatum tui, Textual)
web	Local web UI (undatum web, FastAPI + Jinja2)
mcp	MCP server for AI agents (undatum mcp serve)
langchain	LangChain agent tools
polars, dask	DataFrame interop from the Dataset SDK
s3	S3 cloud storage support (boto3)
gcs	Google Cloud Storage (gs:// / gcs://, gcsfs)
azure	Azure Blob / ADLS (az:// / abfs://, adlfs)
cloud	Multi-cloud storage via fsspec (S3 + GCS + Azure)
postgres, mysql, mssql, clickhouse	Database connectors
frictionless	Full Frictionless Data Package validation
lakehouse	Delta / Iceberg / Lance / DuckLake / Hudi via iterabledata
gis	Geospatial and LiDAR formats
scientific	MATLAB, geophysical, and HDF5 formats
access	Microsoft Access (.mdb / .accdb)
compression	Extra codecs (snappy, brotli, lzo)

pip install "undatum[extract,api]"

After installation both undatum and the shorter data command are available:

undatum --version
undatum headers data.csv
data headers data.csv   # same thing

Shell completion

undatum --install-completion bash   # or zsh / fish
undatum --show-completion bash      # preview without installing

Requirements

• Python 3.9 or greater (CI tests 3.9–3.13)

Install from source

python -m pip install --upgrade pip setuptools wheel
python -m pip install .

Quick start

# Inspect supported formats
undatum formats list --capabilities

# Convert
undatum convert people.csv people.parquet
undatum convert --tagname item data.xml data.jsonl

# Inspect
undatum headers data.jsonl
undatum analyze data.jsonl
undatum stats data.csv
undatum table data.csv --limit 20

# Query
undatum sql "SELECT city, COUNT(*) AS n FROM data GROUP BY city" cities.csv

# Validate and package
undatum validate data.csv --rules rules.yml
undatum package create data.csv --output datapackage.json

# Document
undatum ai doc data.csv
undatum doc data.jsonl --format markdown --output dataset.md

More first-success paths: quick start and the cookbook.

Commands

All commands are available as undatum <command> or via the shorter data alias.

Top-level data commands: convert, extract, analyze, doc (document), stats (profile), validate, schema, schema-bulk, sql, select, search, mask, plot, ingest, tui, web, and the other transform/inspection commands in the CLI reference.

Command groups:

Group	Subcommands
ai	doc, filter, plan, suggest
api	discover, serve, run, openapi
db	query, load, dump
package	create, add-resource, validate
pipeline	run, validate, doc, templates list, templates init
formats	list, describe, export, tables
mcp	serve, tools
examples	list, show, run
plugins	list, info, validate
config	show

undatum convert --help
undatum sql --help

Contributing

Contributions are welcome. See CONTRIBUTING.md and the development docs.

License

MIT License — see LICENSE.

Links

• Documentation
• GitHub
• PyPI
• Changelog
• Issue tracker
• iterabledata (streaming I/O engine)