[spark] Resolve primary-key and bucket-key properties like other identifiers

[zhuxiangyi]

Purpose

primary-key and bucket-key are plain strings in TBLPROPERTIES, so they reach Paimon exactly as
the user typed them and are then matched against the schema exactly. Every other identifier in the
same statement follows spark.sql.caseSensitive — the partition columns in particular, because
Spark's analyzer has already resolved them by the time the catalog sees them. That leaves a single
statement obeying two different rules:

-- accepted: the partition column resolves case-insensitively
CREATE TABLE t (Id INT, Pt STRING) PARTITIONED BY (pt);

-- rejected
CREATE TABLE t (Id INT, A INT) TBLPROPERTIES ('primary-key'='id', 'bucket'='1');
-- IllegalStateException: Table column [Id, A] should include all primary key constraint [id]

-- rejected
CREATE TABLE t (Id INT, A INT) TBLPROPERTIES ('bucket-key'='id', 'bucket'='1');
-- RuntimeException: Field names [Id, A] should contains all bucket keys [id].

Since spark.sql.caseSensitive is false by default, Id and id are the same identifier
everywhere else in Spark, so this is surprising.

Fix. Resolve both properties against the schema in SparkCatalog, honouring the session's
case-sensitivity setting:

• toInitialSchema — for the primary-key list and the bucket-key option value.
• alterTable — for bucket-key only; primary-key cannot reach there because
  validateAlterProperty already rejects altering it. The table is loaded only when the property
  being set is bucket-key, so other SET TBLPROPERTIES calls are unaffected.

Two deliberate choices:

• With spark.sql.caseSensitive=true nothing changes — the exact-match requirement stays.
• A name matching no column is passed through untouched, so Paimon still reports it exactly as
  the user wrote it instead of a rewritten one.

Why this stays on the Spark side. The exact matching in core is correct for Flink: its
identifiers are case-sensitive, so Id and id really are two different identifiers there, and its
primary keys come from the resolved PRIMARY KEY DDL constraint (FlinkCatalog
builder.primaryKey(table.primaryKeys())) rather than an option string. Changing core would break
that. This mirrors how Iceberg handles it — a case-sensitivity flag plumbed through the connector
(SparkUtil.caseSensitive, PartitionSpec.Builder#caseSensitive) rather than exact matching pushed
down into the format. Paimon already has the same precedent on the Spark write path, where
SchemaEvolutionHelper reads conf.caseSensitiveAnalysis.

Side effect worth noting. bucket-key values are now split on , and trimmed before being
stored, which primary-key already did. 'bucket-key' = 'id, sub' previously failed, because
TableSchema#originalBucketKeys splits on , without trimming and then looked for a column named
" sub". It now works, making the two properties consistent.

Not covered here: other options whose values name columns (for example sequence.field) have the
same shape and could be given the same treatment as a follow-up.

Tests

New KeyPropertyCaseResolutionTestBase with concrete suites for Spark 3.2, 3.3, 3.4, 3.5, 4.0 and
4.1 — 7 cases, asserting the stored key uses the column's real spelling and that the table is
actually writable and readable afterwards:

• primary-key / bucket-key resolve like every other identifier
• multi-column and partitioned keys resolve ('primary-key'='id,sub,pt')
• ALTER TABLE SET TBLPROPERTIES resolves bucket-key
• an exact spelling keeps working
• an unknown column still reports the original name
• spark.sql.caseSensitive=true keeps the exact-match requirement

Verified green on Spark 3.2, 3.3, 3.4, 3.5 and 4.0 (7/7 each), and the full paimon-spark-ut module
shows no new failures.