[AhmedMaherTohmay]

Linked issue or discussion

Closes #13

What this changes

Some digitally born Arabic PDFs contain a valid-looking but incorrect /ToUnicode map. Because the mapped characters are valid Unicode, the previous pipeline could return plausible-looking but incorrect text without detecting the corruption.

This PR adds an independent source of glyph identity for composite TrueType fonts:

• reads embedded /FontFile2 programs;
• reads /CIDToGIDMap for CID-to-glyph resolution;
• parses TrueType cmap formats 0, 4, 6, and 12;
• compares single-character /ToUnicode mappings with the embedded font identity;
• overrides a mapping only when their NFKC forms genuinely disagree;
• preserves multi-character mappings such as Arabic ligatures;
• preserves correctly encoded Persian yeh;
• marks pages with widespread mapping disagreements as degraded;
• removes the observed narrow filler glyph only when a usable cmap has already demonstrated that the producer’s map is contradictory;
• validates cmap ranges and CID-map bounds defensively.

The implementation includes focused unit tests and named real-document regression tests.

Output changes

Yes. This fixes silent corruption in Arabic PDFs whose /ToUnicode maps disagree with their embedded TrueType fonts.

doc3.pdf, page 1

Before:

الجريدة الرسمية
لزار ببلمىافمت ػىٍ اػخببر صبلاث رجبي الأػبّي والمسبفزٌٓ

After:

الجريدة الرسمية
استعرض مجلس الوزراء قراره

The corrected output also includes:

قرار بالموافقة على اعتبار صالات رجال الأعمال والمسافرين
قرر مجلس الوزراء في جلسته

A page with widespread disagreement is now reported as degraded, reflecting that its text was recovered by rejecting many of the PDF producer’s own /ToUnicode claims.

doc3.pdf demonstrate the issue and can be provided for corpus verification if their redistribution is permitted.

Testing

• Added unit coverage for TrueType cmap formats and malformed ranges.
• Added coverage for explicit /CIDToGIDMap bounds.
• Added coverage ensuring genuine Persian yeh is preserved.
• Added coverage ensuring narrow glyphs are not dropped without usable TrueType evidence.
• Added named real-document regressions for the incorrect /ToUnicode mappings and filler glyph.

Corpus request

This behavior was discovered using real Arabic government PDFs. Since font behavior varies significantly between PDF producers, expanding the corpus with more redistributable Arabic PDFs would make these decisions safer.

Useful additions would include:

• PDFs from different authoring tools and producers;
• composite TrueType and CFF fonts;
• missing, malformed, or incorrect /ToUnicode maps;
• mixed Arabic, Latin, and numeric text;
• tagged and untagged Arabic PDFs.

A broader corpus will help ensure that a fix for one producer does not silently damage another.

Checklist

• The linked issue has the approved label
• The branch is named fix/13-tounicode-font-arbitration
• cargo test passes
• cargo clippy --all-targets and cargo fmt --check are clean
• Output was checked against the local golden corpus
• The bug fix includes named regression tests

[khalil-Hennara]

Hi @AhmedMaherTohmay I am really sorry, I want be able to merge this pull request, as I mentioned in the issue, that I am gonna work on this issue, as it's would affect the whole pipeline of the project. From the first point I've assumed that all Arabic Pdf, is Type0 Font, and with you file we should now handel TrueType font, this will affect almost all files in the project, and that why I say I want to work on it, but I really appriciate your effort, and the document you provide change a lot. I would invite you to build the comman pdf folder that would be our base benchmark for upgrading this project, I will also share all my test files I've used it to build the system.

[AhmedMaherTohmay]

No worries at all.
My bad, I totally misunderstood your thinking that you were giving me the go ahead for the PR
And yeah it makes total sense though handling TrueType fonts sounds like a huge shift for the whole pipeline so take your time with it, and of course I'd love to help out with the benchmark folder whenever you're ready.
Finally I have attached another file that I believe it would help a lot when migrating to the new pipeline and fixing this issue.
doc5.pdf

[khalil-Hennara]

Thanks @AhmedMaherTohmay I really appreciate you support, I will mention you as soon as possible when gonna work on the test data. Thanks again brother