Skip to content

TIKA-4814 -- OneNote dom-order extraction and follow-ups #3046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tballison merged 1 commit into from
Aug 22, 2026
Merged

TIKA-4814 -- OneNote dom-order extraction and follow-ups #3046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,17 @@ Release 3.3.3 - ???
* MagicDetector now compiles its regular expression once, in the
constructor, instead of recompiling it on every match (TIKA-4796).

* OneNote extraction now follows document order, omits superseded page
revisions, sorts author metadata, extracts embedded object BLOBs, and
bounds malformed-input recursion and file-derived allocations. Parse
warnings and embedded relationship IDs are exposed in metadata. Malformed
or truncated files that cannot be fully parsed, and files whose walk
yields no content, now fall back to the legacy string dump instead of
failing or returning empty output. The legacy MS-ONESTORE walker bounds
its recursion (depth caps plus file-node-list and fragment-chain cycle
guards) and now honors shouldParseEmbedded for embedded file data
(TIKA-4814).

* PDFs whose %PDF- header is preceded by a print-composition job ticket are
no longer detected as text/x-matlab. Up to 50 %% comment or blank lines of
up to 150 characters, and nothing else, may now precede the header; the
Expand Down
15 changes: 12 additions & 3 deletions ...-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/FileNodePtr.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,21 @@ public FileNode dereference(OneNoteDocument document) throws TikaException {
if (nodeListPositions.isEmpty()) {
return null;
}
if (nodeListPositions.get(0) >= document.root.children.size()) {
int rootPosition = nodeListPositions.get(0);
if (rootPosition < 0 || rootPosition >= document.root.children.size()) {
throw new TikaException("Exceeded root child size");
}
FileNode cur = document.root.children.get(nodeListPositions.get(0));
FileNode cur = document.root.children.get(rootPosition);
for (int i = 1, ie = nodeListPositions.size(); i < ie; ++i) {
cur = cur.childFileNodeList.children.get(nodeListPositions.get(i));
if (cur == null || cur.childFileNodeList == null ||
cur.childFileNodeList.children == null) {
throw new TikaException("Missing child file node list");
}
int childPosition = nodeListPositions.get(i);
if (childPosition < 0 || childPosition >= cur.childFileNodeList.children.size()) {
throw new TikaException("Exceeded child file node size");
}
cur = cur.childFileNodeList.children.get(childPosition);
}
return cur;
}
Expand Down
40 changes: 34 additions & 6 deletions ...-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import java.util.List;
import java.util.Locale;

import org.apache.tika.exception.TikaException;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.BitConverter;
import org.apache.tika.utils.StringUtils;

Expand All @@ -41,17 +42,44 @@ public GUID(int[] guid) {
* -EEEEEEEEEEEE}
* @return GUID object parsed from guid bytes.
*/
public static GUID fromCurlyBraceUTF16Bytes(byte[] guid) {
public static GUID fromCurlyBraceUTF16Bytes(byte[] guid) throws TikaException {
int[] intGuid = new int[16];
String utf16Str = new String(guid, StandardCharsets.UTF_16LE).replaceAll("\\{", "")
.replaceAll("-", "").replaceAll("}", "");
for (int i = 0; i < utf16Str.length(); i += 2) {
intGuid[i / 2] =
Integer.parseUnsignedInt("" + utf16Str.charAt(i) + utf16Str.charAt(i + 1), 16);
String utf16Str = new String(guid, StandardCharsets.UTF_16LE);
if (utf16Str.length() != 38 || utf16Str.charAt(0) != '{' ||
utf16Str.charAt(9) != '-' || utf16Str.charAt(14) != '-' ||
utf16Str.charAt(19) != '-' || utf16Str.charAt(24) != '-' ||
utf16Str.charAt(37) != '}') {
throw new TikaException("Invalid GUID string");
}
String hex = utf16Str.substring(1, 37).replace("-", "");
if (hex.length() != 32) {
throw new TikaException("Invalid GUID string");
}
for (int i = 0; i < hex.length(); i += 2) {
int high = asciiHexDigit(hex.charAt(i));
int low = asciiHexDigit(hex.charAt(i + 1));
if (high < 0 || low < 0) {
throw new TikaException("Invalid GUID string");
}
intGuid[i / 2] = (high << 4) | low;
}
return new GUID(intGuid);
}

// Character.digit accepts non-ASCII Unicode digits; GUIDs are ASCII hex only
private static int asciiHexDigit(char c) {
if (c >= '0' && c <= '9') {
return c - '0';
}
if (c >= 'a' && c <= 'f') {
return c - 'a' + 10;
}
if (c >= 'A' && c <= 'F') {
return c - 'A' + 10;
}
return -1;
}

public static int memcmp(int[] b1, int[] b2, int sz) {
for (int i = 0; i < sz; i++) {
if (b1[i] != b2[i]) {
Expand Down
3 changes: 3 additions & 0 deletions ...rosoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ class OneNoteDocument {
Map<ExtendedGUID, Pair<Long, ExtendedGUID>> revisionRoleMap = new HashMap<>();
ExtendedGUID currentRevision = ExtendedGUID.nil();
FileNodeList root = new FileNodeList();
// set when the root file node list could not be fully parsed (e.g. a truncated file);
// the header and any structure parsed so far remain usable
Exception structureParseException;

public OneNoteDocument() {

Expand Down
150 changes: 112 additions & 38 deletions ...icrosoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.time.Instant;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
Expand All @@ -27,10 +28,13 @@

import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;

import org.apache.tika.exception.TikaException;
import org.apache.tika.exception.WriteLimitReachedException;
import org.apache.tika.io.TemporaryResources;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
Expand All @@ -52,6 +56,7 @@
public class OneNoteParser implements Parser {

public static final String ONE_NOTE_PREFIX = "onenote:";
private static final Logger LOG = LoggerFactory.getLogger(OneNoteParser.class);
private static final Map<MediaType, List<String>> TYPES_MAP = new HashMap<>();
/**
* Serial version UID
Expand Down Expand Up @@ -126,53 +131,76 @@ public void parse(InputStream stream, ContentHandler handler, Metadata metadata,
metadata.set(ONE_NOTE_PREFIX + "rgbPlaceholder",
"0x" + Long.toHexString(oneNoteDocument.header.rgbPlaceholder));

Pair<Long, ExtendedGUID> roleAndContext = Pair.of(1L, ExtendedGUID.nil());
OneNoteTreeWalker oneNoteTreeWalker =
new OneNoteTreeWalker(options, oneNoteDocument, oneNoteDirectFileResource,
xhtml, metadata, context, roleAndContext);
Exception structureFailure = oneNoteDocument.structureParseException;
boolean walked = false;
if (structureFailure == null) {
try {
Pair<Long, ExtendedGUID> roleAndContext = Pair.of(1L, ExtendedGUID.nil());
OneNoteTreeWalker oneNoteTreeWalker =
new OneNoteTreeWalker(options, oneNoteDocument,
oneNoteDirectFileResource, xhtml, metadata, context,
roleAndContext);

oneNoteTreeWalker.walkTree();
oneNoteTreeWalker.walkTree();

if (!oneNoteTreeWalker.getAuthors().isEmpty()) {
metadata.set(TikaCoreProperties.CREATOR,
oneNoteTreeWalker.getAuthors().toArray(new String[]{}));
if (!oneNoteTreeWalker.getAuthors().isEmpty()) {
metadata.set(TikaCoreProperties.CREATOR,
sortedValues(oneNoteTreeWalker.getAuthors()));
}
if (!oneNoteTreeWalker.getMostRecentAuthors().isEmpty()) {
metadata.set(
Property.externalTextBag(ONE_NOTE_PREFIX + "mostRecentAuthors"),
sortedValues(oneNoteTreeWalker.getMostRecentAuthors()));
}
if (!oneNoteTreeWalker.getOriginalAuthors().isEmpty()) {
metadata.set(
Property.externalTextBag(ONE_NOTE_PREFIX + "originalAuthors"),
sortedValues(oneNoteTreeWalker.getOriginalAuthors()));
}
if (!Instant.MAX.equals(
Instant.ofEpochMilli(oneNoteTreeWalker.getCreationTimestamp()))) {
metadata.set(ONE_NOTE_PREFIX + "creationTimestamp",
String.valueOf(oneNoteTreeWalker.getCreationTimestamp()));
}
if (!Instant.MIN.equals(oneNoteTreeWalker.getLastModifiedTimestamp())) {
metadata.set(ONE_NOTE_PREFIX + "lastModifiedTimestamp", String.valueOf(
oneNoteTreeWalker.getLastModifiedTimestamp().toEpochMilli()));
}
if (oneNoteTreeWalker.getLastModified() > Long.MIN_VALUE) {
metadata.set(TikaCoreProperties.MODIFIED,
String.valueOf(oneNoteTreeWalker.getLastModified()));
}
walked = true;
} catch (Exception e) {
rethrowIfLimitReached(e);
structureFailure = e;
}
}
if (!oneNoteTreeWalker.getMostRecentAuthors().isEmpty()) {
metadata.set(Property.externalTextBag(ONE_NOTE_PREFIX + "mostRecentAuthors"),
oneNoteTreeWalker.getMostRecentAuthors().toArray(new String[]{}));
}
if (!oneNoteTreeWalker.getOriginalAuthors().isEmpty()) {
metadata.set(Property.externalTextBag(ONE_NOTE_PREFIX + "originalAuthors"),
oneNoteTreeWalker.getOriginalAuthors().toArray(new String[]{}));
}
if (!Instant.MAX.equals(
Instant.ofEpochMilli(oneNoteTreeWalker.getCreationTimestamp()))) {
metadata.set(ONE_NOTE_PREFIX + "creationTimestamp",
String.valueOf(oneNoteTreeWalker.getCreationTimestamp()));
}
if (!Instant.MIN.equals(oneNoteTreeWalker.getLastModifiedTimestamp())) {
metadata.set(ONE_NOTE_PREFIX + "lastModifiedTimestamp", String.valueOf(
oneNoteTreeWalker.getLastModifiedTimestamp().toEpochMilli()));
}
if (oneNoteTreeWalker.getLastModified() > Long.MIN_VALUE) {
metadata.set(TikaCoreProperties.MODIFIED,
String.valueOf(oneNoteTreeWalker.getLastModified()));
if (!walked) {
legacyFallbackDump("OneNote parse failed; falling back to legacy text dump: " +
failureMessage(structureFailure), structureFailure, metadata,
xhtml, oneNoteDirectFileResource);
}
} else if (header.isLegacyOrAlternativePackaging()) {
MSOneStorePackage pkg = null;
try {
AlternativePackaging alternatePackageOneStoreFile = new AlternativePackaging();
//enable streaming deserialization
alternatePackageOneStoreFile.doDeserializeFromByteArray(oneStoreFileBytes, 0);

MSOneStoreParser onenoteParser = new MSOneStoreParser();
MSOneStorePackage pkg =
onenoteParser.parse(alternatePackageOneStoreFile.dataElementPackage);
pkg = onenoteParser.parse(alternatePackageOneStoreFile.dataElementPackage);

pkg.walkTree(options, metadata, xhtml);
pkg.walkTree(options, metadata, xhtml, context);
} catch (Exception e) {
OneNoteLegacyDumpStrings dumpStrings =
new OneNoteLegacyDumpStrings(oneNoteDirectFileResource, xhtml);
dumpStrings.dump();
rethrowIfLimitReached(e);
legacyFallbackDump(
"OneNote FSSHTTPB parse failed; falling back to legacy text dump: " +
failureMessage(e), e, metadata, xhtml,
oneNoteDirectFileResource);
pkg = null;
}
legacyFallbackIfNoContent(pkg, metadata, xhtml, oneNoteDirectFileResource);
} else {
throw new TikaException("Invalid OneStore document - could not parse headers");
}
Expand All @@ -182,6 +210,45 @@ public void parse(InputStream stream, ContentHandler handler, Metadata metadata,

}

private static String[] sortedValues(Set<String> values) {
String[] sorted = values.toArray(new String[0]);
Arrays.sort(sorted);
return sorted;
}

private static void rethrowIfLimitReached(Exception e) throws TikaException, SAXException {
WriteLimitReachedException.throwIfWriteLimitReached(e);
}

private static String failureMessage(Exception e) {
return e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage();
}

// the walk completed but every page dangled - without this a degraded
// file would yield empty output where the dump still finds its text
static void legacyFallbackIfNoContent(MSOneStorePackage pkg, Metadata metadata,
XHTMLContentHandler xhtml,
OneNoteDirectFileResource oneNoteDirectFileResource)
throws TikaException, SAXException {
if (pkg != null && !pkg.hasEmittedContent()) {
legacyFallbackDump("OneNote FSSHTTPB parse produced no content; " +
"falling back to legacy text dump", null, metadata, xhtml,
oneNoteDirectFileResource);
}
}

private static void legacyFallbackDump(String warning, Exception cause, Metadata metadata,
XHTMLContentHandler xhtml,
OneNoteDirectFileResource oneNoteDirectFileResource)
throws TikaException, SAXException {
LOG.warn(warning);
if (cause != null) {
LOG.debug("OneNote parse failure", cause);
}
metadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
new OneNoteLegacyDumpStrings(oneNoteDirectFileResource, xhtml).dump();
}

/**
* Create a OneNoteDocument object.
* <p>
Expand Down Expand Up @@ -215,7 +282,8 @@ public void parse(InputStream stream, ContentHandler handler, Metadata metadata,
* content.
* @return A parsed one note document. This document does not contain any of the binary data,
* rather it just contains
* the data pointers and metadata.
* the data pointers and metadata. A failure while parsing the root file node list is not
* thrown; it is recorded in the returned document's {@code structureParseException}.
* @throws IOException Will throw IOException in typical IO issue situations.
*/
public OneNoteDocument createOneNoteDocumentFromDirectFileResource(
Expand All @@ -227,9 +295,15 @@ public OneNoteDocument createOneNoteDocumentFromDirectFileResource(

if (oneNoteDocument.header.isMsOneStoreFormat()) {
// Now that we parsed the header, the "root file node list"
oneNotePtr.reposition(oneNoteDocument.header.fcrFileNodeListRoot);
FileNodePtr curPath = new FileNodePtr();
oneNotePtr.deserializeFileNodeList(oneNoteDocument.root, curPath);
try {
oneNotePtr.reposition(oneNoteDocument.header.fcrFileNodeListRoot);
FileNodePtr curPath = new FileNodePtr();
oneNotePtr.deserializeFileNodeList(oneNoteDocument.root, curPath);
} catch (TikaException | IOException | RuntimeException e) {
// a truncated or malformed root list is recorded, not thrown, so the
// caller can fall back to the legacy string dump
oneNoteDocument.structureParseException = e;
}
}
return oneNoteDocument;
}
Expand Down
Loading
Loading