Add support for Canonical XML

[decryphe]

First and foremost: It's a great library!

While dabbling in validating signed XML files, I've come to the point at which I need to generate the canonical XML representation. The existing specifications are:

• https://www.w3.org/TR/2001/REC-xml-c14n-20010315
• https://www.w3.org/TR/xml-c14n11/
• https://www.w3.org/TR/2002/REC-xml-exc-c14n-20020718/

As these introduce rather sweeping requirements on how exactly to serialize XML, I'm wondering whether you would consider such a feature for Xot, if you'd want to work on it, or if you'd accept a PR for such a feature?

As the specs require modifications to the XML document (e.g. replacing CDATA sections), what would you envision a hypothetical Xot function would look like to turn an XML document/element into its canonical representation?

Regards

[faassen]

Thank you!

This is where we control serializatoin:

https://docs.rs/xot/latest/xot/output/xml/struct.Parameters.html

The serializer in Xot is hand-written. I've been vaguely playing with the idea of switching to quickxml for parsing/serializing but if that's going to happen it's a while off, and having a hand-written serializer is helpful as we can add just what we need.

Here's the list from https://www.w3.org/TR/xml-c14n11

The document is encoded in UTF-8

Internally this already happens, and is the default behavior

Line breaks normalized to #xA on input, before parsing

This one I'm not sure about and needs to be investigated.

Attribute values are normalized, as if by a validating processor

There's some normalization but needs to be dug into further.

Character and parsed entity references are replaced

This already happens during parsing.

CDATA sections are replaced with their character content

Happens already during parsing. So we shouldn't need to modify the XML document as CDATA sections are normalized into plain text nodes anyway. This information is not retained, though you can control CDATA section generation using the serializer (normally they're never serialized).

The XML declaration and document type declaration are removed

Can be controlled by serialization parameters; I believe it's even the default.

Empty elements are converted to start-end tag pairs

This needs another parameter, off by default.

Whitespace outside of the document element and within start and end tags is normalized

Within start and end tags this is happening already. Outside the document element I'm not sure about; I thought
text nodes weren't even allowed there anyway?

All whitespace in character content is retained (excluding characters removed during line feed normalization)

I think this happens.

Attribute value delimiters are set to quotation marks (double quotes)

This is the behavior for serialization.

Special characters in attribute values and character content are replaced by character references

I don't know; it doesn't seem to define what "special characters" are? Might need a serialization parameter.

Superfluous namespace declarations are removed from each element

This is probably easiest to do with a document transformation that removes namespace prefixes.

This does part of the work already, but we may want to have a method that deduplicates and removes unused prefixes, which I don't think we do yet:

https://docs.rs/xot/latest/xot/struct.Xot.html#method.deduplicate_namespaces

Default attributes are added to each element

We don't do anything with default attributes but we don't handle DTDs anyway, so maybe there cannot be such
default attributes.

Fixup of xml:base attributes [C14N-Issues] is performed

We don't do that as far as I know.

Lexicographic order is imposed on the namespace declarations and attributes of each element

We maintain definition order, though we do handle namespace declarations before attributes. So this needs an additional parameter.

Conclusion

We're quite far along already. But we need more serialization parameters to get there. Then we should also add a "canonical_string" method or something like that which sets up all the parameters in the right way to get canonical content.

PRs are welcome! You could tackle each new parameter separately slowly building this up.

https://github.com/faassen/xot/blob/main/src/output/xml_serializer.rs is where serialization happens. One of the harder things involves ordering the attributes and such. I think the easiest way to do this is to reorder attributes in the document itself. In fact we need a "canonicalization sweep" anyway to clean up unused prefixes too. An alternative is some kind of buffering serializer which collects all prefix and attribute outputs of an element and then sends them to the underlying serialization code in a different order.

[decryphe]

Wow, that's quite the analysis!

Reading all of it, I do agree with your conclusion. Many of the steps can be implemented individually and some seem quite easy indeed. I should actually be able to find some time to work on these things, so hopefully I can make some PRs over the next couple weeks/months.

I'll update the initial post in this issue to include a checklist of what's needed and what's already there, so this issue can be used to track progress.