[core] Add field-id.one-based option for strictly positive Iceberg field ids

[vbabenkoru]

Purpose

Paimon starts field ids at 0. The Iceberg ecosystem starts them at 1. Because of this, Iceberg metadata for a Paimon table contains a field with id 0. Some external Iceberg readers reject these tables. For example, Snowflake refuses them. This also requires workarounds elsewhere. When a partition column has field id 0, the REST committer creates a dummy schema or falls back to partition evolution.

This PR adds an opt-in table option, field-id.one-based. It shifts every field id by one when the table is created. This includes top-level and nested fields. The table's id space then matches what Iceberg-native readers expect from the start.

CREATE TABLE t (...) WITH (
    'field-id.one-based' = 'true',
    'metadata.iceberg.storage' = 'rest-catalog',
    ...
);

Design points:

• Creation-time only, immutable. Field ids are stored in data files (Parquet footers) and Iceberg metadata. Changing the base of an existing table would break column resolution. The option is @Immutable. It is applied once in SchemaManager.createTable. Existing tables keep their ids.
• A shift, not a reassignment. The new ShiftFieldId visitor moves the whole id space by a fixed offset. It keeps the relative order and any gaps. This differs from ReassignFieldId, which renumbers ids without gaps. Schema evolution then continues from the shifted highest field id.
• Default unchanged. If the option is not set, ids remain zero-based. The result is byte-for-byte identical to today.

Tests

• FieldIdOneBasedTest (paimon-core): checks that the default stays zero-based, nested ids are shifted, schema evolution continues from the shifted ids, and the option is create-time-only and immutable.
• IcebergFieldIdOneBasedCompatibilityTest (paimon-iceberg): runs end-to-end on a primary-key DV table with Iceberg format version 3. It checks that the Paimon schema, Iceberg metadata, and Parquet footers all use the same strictly positive ids. It verifies them with the Iceberg reader. It also includes a control test showing that the default remains zero-based.

API and Format

New optional table option field-id.one-based. Its default is false, and it is immutable. Existing tables and the default id assignment do not change.

Documentation

The option is documented in its description, which is included in the generated configuration docs.

---

AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.

[JingsongLi]

Although the option claims to be create-time-only or immutable, the general ALTER validation only checks for immutability when an existing snapshot is present (SchemaManager:354). Consequently, after table creation but before the first snapshot, one can toggle field-id.one-based between false and true; while the field IDs themselves are not reordered, the schema option is overwritten, resulting in a permanent inconsistency where the configuration specifies "one-based" indexing while the actual schema still includes ID 0. New tests have even codified this behavior. Changes to the actual value should be unconditionally prohibited, just as they are for the type attribute.

[vbabenkoru]

Thank you, fixed.
ALTER validation now rejects the value regardless of snapshots, just like type. Setting a different effective value throws an error. Removing the option from a table created with one-based field ids also throws an error.
Setting the current effective value again is still allowed. For example, you can explicitly set false on a table that never set the option. This is allowed because tools often reapply complete option sets.
The test for the old behavior now checks that both changes are rejected.