Skip to content

Pull requests: huggingface/tokenizers

New pull request New
141 Open 1,090 Closed
141 Open 1,090 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Respect BPE cache_capacity when reading JSON
#2420 opened Sep 17, 2026 by LysandreJik Member Loading…
chore(python): newlines in __repr__
#2418 opened Sep 17, 2026 by SBrandeis Contributor • Draft
feat: Implement tokenize() / decode_tokens()
#2417 opened Sep 17, 2026 by SBrandeis Contributor Loading…
6
feat(python): composition
#2414 opened Sep 16, 2026 by SBrandeis Contributor • Draft
Fix custom prefixes in BertWordPieceTokenizer
#2413 opened Sep 16, 2026 by HaokaiDing Loading…
1
1
feat: encode_special_tokens flag
#2411 opened Sep 16, 2026 by SBrandeis Contributor Loading…
fix(ci): harden GitHub Actions workflows (#2119)
#2410 opened Sep 15, 2026 by hf-security-analysis Bot Loading…
feat(python): add Arrow string batch tokenization
#2400 opened Sep 10, 2026 by granthamtaylor Loading…
3
chore(post-process): remove polymorphism + encode_into
#2399 opened Sep 10, 2026 by SBrandeis Contributor • Draft
2
bpe: do not slice merge tokens at a byte offset that may not be a boundary
#2398 opened Sep 10, 2026 by apollo-2006 Loading…
2
Fix BERT normalization of CJK Extension E boundary characters
#2395 opened Sep 8, 2026 by mottopanikeiku Loading…
fix: Unigram prune_sentence_pieces uses alternatives[id].len() not alternatives.len()
#2394 opened Sep 8, 2026 by VirajMishra1 Loading…
1
fix(python): forward unk_id in SentencePieceUnigramTokenizer
#2393 opened Sep 7, 2026 by Con-Benksl Loading…
1
draft release
#2390 opened Sep 3, 2026 by ArthurZucker Collaborator • Draft
1
fix(bpe): drop heap entries with non-positive pair_counts before u64 cast
#2389 opened Sep 3, 2026 by okxint Loading…
1
perf(decode): index the vocab slab by id, not through the MPHF slot
#2387 opened Sep 3, 2026 by ArthurZucker Collaborator Loading…
1
fix: correctly decode numpy UCS-4 arrays instead of treating as UTF-8
#2386 opened Sep 3, 2026 by VirajMishra1 Loading…
docs: rewrite the README for the crate split
#2385 opened Sep 2, 2026 by ArthurZucker Collaborator Loading…
v1.0.0-rc0: the train/encode split, repacked into 53 commits
#2384 opened Sep 2, 2026 by ArthurZucker Collaborator Loading…
perf(normalizers): fast-path BertNormalizer for non-CJK input
#2383 opened Sep 2, 2026 by bsachart Loading…
2
Fix UnicodeScripts pre-tokenizer dropping leading spaces
#2382 opened Sep 2, 2026 by Kayvan-Zahiri Loading…
perf(parallelism): default the pool to physical cores, not SMT siblings
#2380 opened Sep 2, 2026 by ArthurZucker Collaborator Loading…
perf(train): select top-k in the trainers instead of ordering everything
#2379 opened Sep 2, 2026 by ArthurZucker Collaborator Loading…
perf(word-cache): scan the tag window with shrn, not an emulated movemask
#2377 opened Sep 2, 2026 by ArthurZucker Collaborator Loading…
perf(pre_tokenizers): eliminate per-token heap allocations in ByteLevel decoder
#2376 opened Sep 1, 2026 by bsachart Loading…
3
ProTip! Updated in the last three days: updated:>2026-09-14.