Skip to content

[format] Fix Parquet delta length reads across vector batches - #9366

Open
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-dlba-leng
Open

[format] Fix Parquet delta length reads across vector batches#9366
ArnavBalyan wants to merge 1 commit into
apache:masterfrom
ArnavBalyan:arnavb/fix-dlba-leng

Conversation

@ArnavBalyan

Copy link
Copy Markdown
Member

Purpose

  • Parquet delta encoding stores variable string length based on position on encoded page.
  • For 1 page -> many vector batches, this incorrectly uses the output position had been reset and causes later strings to be truncated.
  • Ensure we can correctly track the variable length string
  • This is a bug fix ported from Spark

Tests

  • UT

@ArnavBalyan

Copy link
Copy Markdown
Member Author

cc @JingsongLi thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant