Postbag 💼

Postbag is a compact binary serde codec for Rust that keeps the Rust type system fully intact and has support for backwards and forwards compatibility built in. This is also known as schema evolution: data written with one version of your types remains readable with another.

An illustrated overview is available at remoc.rs/postbag.

Quick start

Normally you will want to use to_full_vec and from_full_slice:

use serde::{Serialize, Deserialize};
use postbag::{to_full_vec, from_full_slice};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct Person {
    name: String,
    age: u32,
}

let person = Person { name: "Alice".to_string(), age: 30 };

let bytes = to_full_vec(&person)?;
let restored: Person = from_full_slice(&bytes)?;

assert_eq!(person, restored);
# Ok::<(), postbag::Error>(())

Two data format variants

Postbag Full writes each field with its, optionally numbered, identifier and the length of its value, which is what lets fields be added, removed and reordered.

Postbag Slim writes the values and nothing else, in declaration order, which is smaller but only allows add struct fields and enum variants at the end.

Both variants use variable-length integer encoding to save space.

Start with Full and use Slim when you need minimal size and can accept less compatibility.

The wire formats are specified separately:

• Postbag Full format 1.0
• Postbag Slim format 1.0

Backwards and forwards compatibility

As usual a field a reader expects but does not receive takes its #[serde(default)], and a variant it does not know needs a #[serde(other)] fallback.

The following changes to your types are supported:

Change to your types	Full	Slim
Structs
Add a field	anywhere	at the end
Remove a field	anywhere	at the end
Rename a field	when numbered	always
Reorder fields	yes	no
Enums
Add a variant	anywhere	at the end
Remove a variant	anywhere	at the end
Rename a variant	when numbered	always
Reorder variants	yes	no
Change between struct variant and newtype variant holding a struct	yes	yes
Change between tuple variant and newtype variant holding a tuple	yes	yes
Change between unit variant and empty tuple variant	yes	yes
Change between unit variant and empty struct variant	yes	no
Field types
Wrap in Recoverable	struct fields and newtype variants	no
Change bool to Option<Recoverable<T>>	struct fields	no
Wrap in or unwrap from a newtype struct	yes	yes
Change between tuple, tuple struct and fixed-size array	yes	yes
Change between Vec<(K, V)> and a map	yes	yes
Size	small	even smaller

Recoverable values

When a value fails to deserialize, the error normally aborts the whole deserialization. An incompatible change to one type thus renders every enclosing value undecodable as well.

When using Postbag Full, a field annotated with #[serde(with = "postbag::recoverable")] confines a deserialization failure to it. The rest is deserialized as usual and the value is replaced by its Default.

use serde::{Serialize, Deserialize};

# #[derive(Default, Serialize, Deserialize)]
# struct Details { size: u32 }
#[derive(Serialize, Deserialize)]
struct Data {
    name: String,
    #[serde(with = "postbag::recoverable")]
    details: Details,
    count: u16,
}

Should Details change incompatibly, name and count still deserialize correctly and details becomes Details::default().

The attribute is equivalent to wrapping the field type in Recoverable<Details>, and both encode the same as the plain field type. Custom recovery policies are also available; see the recoverable module for details.

Numbered identifiers

A struct field or enum variant renamed to _0 through _59 is encoded as a single byte instead of its name. This allows significant space savings in Full mode, while still providing full backwards and forwards compatibility.

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct CompactData {
    #[serde(rename = "_3")]
    my_field: u32,
    #[serde(rename = "_15")]
    another_field: String,
    // Regular field names work normally.
    normal_field: bool,
}

#[derive(Serialize, Deserialize)]
enum CompactEnum {
    #[serde(rename = "_0")]
    MyLongVariantName,
    #[serde(rename = "_1")]
    AnotherLongVariantName(u32),
    #[serde(rename = "_2")]
    YetAnotherVariant {
        // Fields of struct variants can be numbered as well.
        #[serde(rename = "_0")]
        my_field: u32,
    },
    // Regular variant names work normally.
    NormalVariant,
}

Numbering is optional and can be mixed with names in the same type. A name that is not of the form _n is written out as a string.

The identifier is what a reader matches on, so changing the id of a field or variant is a breaking change, and an id that has been retired must never be given to a different field or variant.

The compact module provides smaller representations of common standard library types, which would otherwise spell out their field and variant names.

Unsupported serde attributes

As a binary format Postbag cannot be used with serde's untagged, internally tagged and flatten attributes.

Nesting depth limit

Serialization and deserialization of nested data is recursive, so deeply nested data consumes stack space. To prevent untrusted input from aborting the process by overflowing the stack, the nesting depth is limited to cfg::DEFAULT_DEPTH_LIMIT (128) and exceeding it fails with Error::RecursionLimit.

This only becomes relevant for recursive types, since the nesting depth of a non-recursive type is bounded by the type itself. Unknown fields are skipped by length rather than parsed, so unknown data cannot cause recursion.

Fast compile mode (for development use)

Postbag supports an optional fast compile mode that reduces compilation time at the cost of buffering struct field data in memory during deserialization, instead of streaming it directly from the reader.

Enable it by setting the postbag_fast_compile cfg flag:

RUSTFLAGS="--cfg postbag_fast_compile" cargo build

Or add it to your .cargo/config.toml for development:

[build]
rustflags = ["--cfg", "postbag_fast_compile"]

This flag is intended for development use only; production builds must not use it.

Limitation: in fast compile mode, fields are read positionally, so adding or removing a struct field anywhere but at the end is not supported. Adding and removing fields at the end continues to work. Serialization is unaffected, so an endpoint built with this flag interoperates with one built without it as long as both use the same types.

Origins

Postbag started as a fork of postcard with the intent to add forward and backward compatibility to the serialized data format. While postcard provides excellent performance and compact encoding, postbag extends this foundation to support schema evolution and data format compatibility across different versions of your applications.

License

Postbag is licensed under the Apache 2.0 license.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Postbag by you, shall be licensed as Apache 2.0, without any additional terms or conditions.