Support Hudi Variant reads via Arrow-rs

[wirybeaver]

Background

Apache Hudi supports a Variant logical type represented in Avro as a record with logicalType = "variant" and encoded child fields such as metadata, value, and optionally typed_value. The hudi-rs read path is already Arrow-native, and arrow-rs/parquet-rs exposes Variant support through parquet::variant::{VariantType, VariantArray} behind the parquet/variant_experimental feature.

Proposal

Add read compatibility for Hudi Variant columns by leaning on arrow-rs rather than introducing a separate hudi-rs Variant model.

Scope

• Enable the parquet-rs Variant feature in the workspace dependency.
• Detect Hudi Avro records with logicalType = "variant" during Avro-to-Arrow schema conversion.
• Represent Variant columns as Arrow Struct fields carrying the arrow.parquet.variant extension metadata.
• Preserve existing read APIs that return Arrow RecordBatch values.
• Verify that returned columns can be converted with parquet::variant::VariantArray::try_new.
• Cover Parquet base-file reads and projection of Variant columns.

Out of scope

• Hudi write support for Variant columns.
• New public hudi-rs-specific Variant APIs.
• SQL functions for querying nested Variant values.

Validation

The implementation should pass targeted Avro/Parquet tests, workspace Rust clippy/format checks, all Rust workspace tests, and the MSRV workspace check.