bjdata

BJData (draft 4 specification) implementation in Dart.

Encoding/decoding of BJData to/from Dart objects in an API based on the dart:convert package.

Usage

import 'package:bjdata/bjdata.dart';

void main() {
    final List<int> encoded = bjdataEncode({
        'hello': 'world',
        'pi': 3.14159,
        'happy': true,
        'list': [1, 0, 1],
        'binary': ByteData.sublistView(Uint8List.fromList([0xDE, 0xAD, 0xBE, 0xEF])),
        'nothing': null,
    });
    
    final decoded = bjdataDecode(encoded);

    print(bjdataBlockNotation(null)); // [Z]
    print(bjdataBlockNotation(true)); // [T]
    print(bjdataBlockNotation(false)); // [F]
    print(bjdataBlockNotation(42)); // [U][42]
    print(bjdataBlockNotation(3.14)); // [D][3.14]
    print(bjdataBlockNotation('Hello, world!')); // [S][U][13][Hello, world!]
    print(bjdataBlockNotation([1, 2, 3])); // [[][U][1][U][2][U][3][]]
    print(bjdataBlockNotation({'foo': 1, 'bar': 2})); // [{][U][3][foo][U][1][U][3][bar][U][2][}]
}

Structure-of-Arrays

Draft 4 adds Structure-of-Arrays (SoA) containers, which store a table of uniform records as a payload-less schema followed by tightly packed binary data, instead of repeating every field name in every record.

Nothing to switch on: any list that is a uniform table of records is packed automatically:

final records = [
    {'id': 1, 'name': 'Alice', 'active': true},
    {'id': 2, 'name': 'Bob', 'active': false},
];

bjdataEncode(records); // Structure-of-Arrays container
bjdataDecode(encoded); // List<Map<String, Object?>>

Configuration

Encoding takes a BjdataConfig, so the settings are declared once rather than threaded through call sites:

const config = BjdataConfig(
    version: BjdataVersion.draft4,  // specification revision to stay within
    soa: BjdataSoaLayout.rowMajor,  // how tables are packed
    multiDimensional: true,         // may a dimension array be used as a count
    compactTypes: true,             // may a value be written as a smaller type
);

bjdataEncode(value, config: config);

// or set it once on a codec
const codec = BjdataCodec(config: BjdataConfig.draft3);
codec.encode(value);

Decoding accepts everything this library understands, so none of these affect it.

Layout

soa selects how the payload is arranged, or turns the packing off:

BjdataSoaLayout	Marker	Payload
rowMajor (default)	[$	Each record contiguous
columnMajor	{$	Each field contiguous
off	—	Plain array of objects

final table = [
    {'a': 1, 'b': 2},
    {'a': 3, 'b': 4},
];

bjdataEncode(table); // payload 1 2, 3 4
bjdataEncode(table, config: const BjdataConfig(soa: BjdataSoaLayout.columnMajor)); // 1 3, 2 4
bjdataEncode(table, config: const BjdataConfig(soa: BjdataSoaLayout.off)); // array of objects

Both layouts carry the same schema and the same number of payload bytes, so the choice is about access rather than size: a column is one contiguous run for a reader that walks fields rather than records. All three decode to the same List of record Maps.

Nested lists become an N-dimensional container, and come back with the same nesting:

final grid = [
    [{'x': 0}, {'x': 1}, {'x': 2}],
    [{'x': 3}, {'x': 4}, {'x': 5}],
];

bjdataEncode(grid);    // [${x:U}#[U2 U3] followed by six packed records
bjdataDecode(encoded); // the same 2x3 nesting of records

Dimension-array counts (#[Nx Ny ...]) are a draft 3 construct, and are also what a rectangular nesting of typed rows is written with. Set multiDimensional: false for a consumer that reads a container counted by an integer but not one counted by a dimension array; each inner table is then packed on its own inside an ordinary array, so the values are unchanged either way.

Compatibility

version is a ceiling on what may be written. BjdataVersion.draft3 never writes packed tables, whatever layout is asked for, since a draft 3 reader cannot parse them:

bjdataEncode(records, config: BjdataConfig.draft3); // array of objects
bjdataEncode(records, config: BjdataConfig.draft4); // packed table, as the defaults do

SoA is the only draft 4 addition this library emits — extension types (E) are not implemented — so BjdataConfig.draft3 and BjdataConfig(soa: BjdataSoaLayout.off) currently produce the same bytes. Prefer draft3 when the reason is the consumer's age, so that later revisions stay capped too.

What gets packed

A list is packed only when every record agrees; anything else is written as a plain array of objects:

Packed	Left as a plain array
Two or more records with the same field names	A single record, or an empty list
Fields with one type across every record	A field that is null in some records only
int fields (narrowest marker that fits)	A field mixing int and double
double, bool, all-null fields	Fields holding TypedData
String fields (dictionary, or char if single)	Records with non-String keys
BigInt fields (high-precision dictionary)	Ragged nested lists
Nested objects and equal-length arrays	Empty or differently sized nested arrays

A single record is never packed, because its schema costs about as much as the object it would replace.

Compact types

The encoder picks the type marker that stores a value in the fewest bytes, rather than the one its Dart type implies. Values are always preserved exactly; what can change is the Dart type they decode back to.

bjdataEncode([for (var i = 0; i < 10000; i++) i % 200]);
// [$U#… rather than a generic array: 20002 -> 10007 bytes, decodes as a Uint8List

bjdataEncode(Uint32List.fromList(small));   // written as uint8, 75% smaller
bjdataEncode(Float64List.fromList(halves)); // written as float16, 50% smaller
bjdataEncode(1.0);                          // float16, 9 bytes -> 3

A type is only changed when that actually saves bytes, so a positive Int64List is not swapped for uint64 and a tie leaves the encoding alone. Floats narrow only when every value survives the narrower type unchanged, so [0.1, 0.2] stays float64. Typed data stays a strongly-typed array: passing a Uint32List is itself a request for one, and a generic array only beats it on a handful of elements. Plain lists have both forms measured, which is why a list of small numbers packs while [1, 2, 3, 1000000] does not — a strong type must be wide enough for its largest value and pays that width throughout.

byte is never re-chosen. It is already the narrowest width, so nothing could be gained, and the specification gives it a meaning of its own.

Pass BjdataConfig(compactTypes: false) when the decoded Dart types matter as much as the values.

	off	on
List<int>, 10k small values	20002	10007
Uint32List, all values < 256	20007	5007
Float64List of halves	40007	20007
Float64List of real measurements	40007	40007
200×50 matrix	80010	40010

Cost

Detection is a single pass over the list, and it is cheaper than what it saves. Encoding 100,000 records of five fields on a VM build:

	Time	Size
soa: BjdataSoaLayout.off	70 ms	5.5 MB
default (rowMajor)	51 ms	2.4 MB
default, table rejected on the last field	92 ms	5.5 MB

Packing is faster than not packing, because roughly half as many bytes are written. The worst case — a list that looks uniform until the very last field and then falls back — costs about 30% over a plain encode. A list that is obviously not a table (its first element is not a record) is rejected immediately and costs nothing measurable.

Tool

dart pub global activate bjdata

# Show help
bjdata -h
# or
dart pub global run bjdata -h

# Encode a JSON file to BJData
bjdata encode input.json output.bjda

# Options: --draft=N, --no-soa, --column-major, --no-nd
bjdata encode input.json output.bjd --column-major
bjdata encode input.json output.bjd --draft=3

# Decode a BJData file to JSON
bjdata decode input.bjd output.json

# Pretty-print a JSON file in BJData block notation
bjdata print input.json

# stdin/stdout can be used instead of filenames
cat input.json | bjdata encode
cat input.bjd | bjdata decode
echo -n "[1, 2, 3]" | bjdata print

Types

Decoding BJData to Dart

• N-dimensional arrays (#[Nx Ny ...]) decode to nested lists, with the innermost axis kept as the typed list. Both row-major and column-major (#[[Nx Ny ...]]) orderings are read; a column-major payload is reordered so that it reads the same way. Only the row-major form is written.
• Extension types (E) are not supported and are rejected with a FormatException.

BJData Type	Marker	Dart
null	Z	null
true	T	true
false	F	false
int8	i	int
uint8	U	int
int16	I	int
uint16	u	int
int32	l	int
uint32	m	int
int64	L	int
uint64	M	int *
float16	h	double
float32	d	double
float64	D	double
byte	B	int
char	C	String
string	S	String
huge	H	BigInt
array	[]	List
array[byte]	[$B	ByteData
array[int8]	[$i	Int8List
array[uint8]	[$U	Uint8List
array[int16]	[$I	Int16List
array[uint16]	[$u	Uint16List
array[int32]	[$l	Int32List
array[uint32]	[$m	Uint32List
array[int64]	[$L	Int64List
array[uint64]	[$M	Uint64List
array[float16]	[$h	Float32List
array[float32]	[$d	Float32List
array[float64]	[$D	Float64List
object	{}	Map
array[T] N-D	#[	Nested List of T ‡
soa[rows]	[${	List<Map> †
soa[columns]	{${	List<Map> †

* Warning: int in Dart is a signed 64-bit integer. uint64/M values are decoded as int64 (i.e. values greater than 9223372036854775807 are decoded as negative values).

Encoding Dart to BJData

Dart	Marker	BJData Type
null	Z	null
bool	TF	bool
int	UiuImlML	int *
double	D	float64
String	S	string
BigInt	H	huge
List	[]	array
ByteData	[$B	array[byte] **
Int8List	[$i	array[int8]
Uint8List	[$U	array[uint8]
Int16List	[$I	array[int16]
Uint16List	[$u	array[uint16]
Int32List	[$l	array[int32]
Uint32List	[$m	array[uint32]
Int64List	[$L	array[int64]
Uint64List	[$M	array[uint64]
Float32List	[$d	array[float32]
Float64List	[$D	array[float64]
Map	{}	object
nested typed	#[	array[T] N-D ‡
List<Map>	[${	soa (row-major) †
List<Map>	{${	soa (column-major) †

‡ A rectangular nesting of typed lists of the same type and length, such as a List<Float64List>, is written as one N-dimensional array. Only typed data is packed this way, matching how a flat list is written, so a List<List<double>> stays a nested array. Pass BjdataConfig(multiDimensional: false) to opt out.

† See Structure-of-Arrays. The layout, the specification revision and N-dimensional packing are all chosen with config:. Both layouts hold the same records, so both decode to the same list of maps.

* int values are encoded using the smallest integer type possible, favouring unsigned types.

** ByteData is recommended for encoding "binary data" as per the BJData specification (as this may affect how the data is parsed in other libraries). Converting from Uint8List to ByteData can be done using ByteData.sublistView(list).

Web

The package can be used in web applications, however it is affected by the JavaScript number peculiarities.

• int values greater than 9007199254740991 (2^53 - 1) may lose precision when encoding/decoding.
• double values without a fractional part will be encoded as int (important if consumer strictly expects a double value).
• Int64List and Uint64List are not supported on web, so array[int64] and array[uint64] will be decoded as List<int>.