feat: Implement tokenize() / decode_tokens() - #2417

Merged

SBrandeis merged 4 commits into

feat/train_encode_splitfrom

feat/python-tokenize

Sep 18, 2026

Commits on Sep 17, 2026

Implement tokenize() / decode_tokens()
SBrandeis
committed
a8f40ee
rm some tests
SBrandeis
committed
0b3992d
Update bindings/python/src/tokenizer.rs

SBrandeis
and
ArthurZucker
authored
fb7e295
accept encoding in decode
SBrandeis
committed
46e0ade