[wtsnz]

While adding aggregate support to AshSQLite, I didn't know how much of Ash's behaviour a data layer is actually meant to support, or how far SQLite was from AshPostgres. I built an aggregate conformance suite in AshSQL (wtsnz/ash_sql#9) to map that out. This PR turns it into a data-layer conformance suite hosted in Ash.

It is two things:

1. A master list of what a data layer does for Ash users. It starts from the basics (read, create, update, destroy, types) and builds up through filtering, sorting and pagination, relationships, aggregates, writes, transactions, multitenancy and authorization.
2. A report per data layer against that list, saying what works, what is rejected, what is wrong, and who owns each gap.

Postgres is a comparison implementation, not the definition of correct. Expected answers come from Ash's documentation and tests, or from decisions recorded in GAPS.md and SEMANTICS.md.

Start with FEATURES.md

conformance/FEATURES.md lists 80 features in 10 levels. Each feature is backed by scenarios that run through Ash's public API, and every scenario belongs to exactly one feature. For each data layer the report shows one of:

• works;
• partial;
• not supported (a documented rejection);
• broken;
• open question (awaiting a semantic decision);
• untested.

Each non-working feature links to the gap behind it, with an owner: Ash, AshSQL, AshSQLite or AshPostgres.

A "claims versus results" section compares each data layer's can?/2 claims with what happened, and flags three cases:

• features advertised but not working;
• features working without being advertised, usually because Ash provides them;
• features not advertised, yet not rejected either, that return wrong answers.

	Pass	Unsupported	Known defect	Unresolved
SQLite	231	42	19	5
PostgreSQL	265	0	32	5

That's 302 scenarios and 599 adapter checks. Four features are listed as untested so the specification stays complete:

• union all and intersection;
• pagination while data changes;
• managing relationships on write;
• isolation between concurrent transactions.

Data layers without reviewed expectations

mix conformance.survey ADAPTER runs every scenario an integration can host and classifies each result against the intended answer: works, rejected, wrong, crashed or open question. The report is marked unreviewed.

surveys/features-ets.md is Ash's own ETS data layer surveyed this way: 252 of 297 work. It needs no Ecto, repository or SQL. Adapters in other repositories can join through config, and supply their own expectations and resource configuration without editing shared code.

Comparing Ash main with the unreleased work

CONFORMANCE_DEPS=upstream swaps all three adapters to ash-project main, with a separate lock file. mix conformance.compare_surveys then diffs two surveys feature by feature.

	Works on main	Works on the pinned work
SQLite	117	231
PostgreSQL	same	same

On SQLite, 22 features change between the two. Postgres is identical, so the aggregate extraction changes nothing there. The comparison also surfaced a regression in ash_sqlite#232 on its own: sorted, paginated reads with a to-many filter return duplicates. A CI job posts this comparison to the job summary on every push, without blocking.

How results are kept honest

• Three seed orders. Every scenario runs with fixtures seeded forward, in reverse and rotated. A result that depends on row order never passes, and a defect that does is pinned for each order. This turned one "pass" and six Postgres defects into order-dependent results.
• Strict comparison. 2 and 2.0 are different results.
• Exact gaps. Gaps pin the exact wrong answer, or the error class and a narrow message. A gap that starts passing fails until someone reviews it and marks it supported.
• Fallback evidence. Where a scenario names the Ash fallback it expects, the runner records data-layer query counts alongside the result.
• Environment. Reports record the runtime, dependency, database and repo settings. Capabilities can depend on these; for example, AshSQLite advertises transactions only when its repo enables them.

Notable findings

• AshPostgres returns another tenant's record from a skipped upsert when return_skipped_upsert? is set.
• Postgres list aggregates without a sort include nil, although include_nil? defaults to false. This predates the extraction.
• SQLite: through relationships load wrong data, because Ash only warns at compile time. Many-to-many load limits apply across all parents instead of per parent.
• SQLite: upsert conditions crash, and decimals lose precision.
• Open question: Ash's docs say true or nil is nil; every SQL data layer returns true.

Layout

conformance/lib is organised by responsibility:

• contracts/ holds the feature catalog, expectations, gaps and claims;
• scenarios/ has one file per feature level, with aggregates/ by topic;
• resources/, fixtures/, adapters/ (one file per adapter), sql/, report/ and benchmark/ hold the rest.

AUTHORING.md covers adding scenarios and adapters.

Benchmarks

mix conformance.bench runs seven warm, single-client read workloads, including an unrelated-rows selectivity experiment. Each answer is checked before timing, and results include raw samples, query counts and run metadata. See BENCHMARKS.md.

Dependencies

Dependency	Pin	Why
Ash	this checkout (override: true)	Every adapter compiles against the Ash under test
AshSQL	wtsnz/ash_sql 0985b9f (override: true)	Unreleased aggregate extraction (wtsnz/ash_sql#3)
AshSQLite	wtsnz/ash_sqlite 46a4b86	Grouped aggregate support (ash-project/ash_sqlite#232)
AshPostgres	ash-project/ash_postgres 945073e4	Current main

mix.upstream.lock pins the upstream set. CONFORMANCE_ASH_*_PATH points at local checkouts for development; CI never sets it.

Validation

On Elixir 1.20.0 / OTP 29, PostgreSQL 17.5 and SQLite 3.53.4:

• 680 conformance tests pass (377 SQLite-only, 382 Postgres-only).
• Format, strict Credo, --warnings-as-errors and Dialyzer pass.
• Ash's 6 dispatch tests pass.
• Generated documents and the ETS survey are verified in CI.
• Both benchmark smokes validate their answers.
• CI passes for SQLite, Postgres, core dispatch and the stack comparison.
• Ash CI's audit job fails on an advisory for mint 1.10.0 in Ash's root mix.lock, which this PR doesn't change.

[wtsnz]

Superseded by #7, which carries this PR's commits unchanged together with the rest of the stack as one PR.