grouptok

A translation-grouped tokenizer: subwords that translate each other share a group

grouptok trains a byte-level BPE tokenizer on sentence pairs. It then aligns the subwords of each pair with a neural word aligner and puts subwords that are often aligned into one group. Each token becomes a (group, member) pair:

 " house" -> (412, 0)      " maison" -> (412, 1)      " Haus" -> (412, 2)
 " king"  -> (977, 0)      " roi"    -> (977, 1)

A language model can then predict the group first and the member second, so translations share most of their parameters. The tokenizer itself is an ordinary BPE tokenizer (it saves a standard tokenizer.json, so it also loads in transformers), plus a groups.json that holds the grouping.

Install

pip install grouptok            # load and use a trained tokenizer (needs only `tokenizers`)
pip install "grouptok[train]"   # train one (adds torch and transformers for the aligner)
pip install "grouptok[train,datasets]"   # and train on Hugging Face datasets directly (see Sources)

Training a tokenizer

Training needs the train extra. It runs on a GPU when one is available.

from grouptok import GroupedTokenizer, TokenizerConfig

pairs = [("The king said unto the people", "Le roi dit au peuple"), ...]   # (one language, the other)
special = ("<|endoftext|>", "<|im_start|>", "<|im_end|>")   # pad, bos, eos; none unless you set them
tok = GroupedTokenizer.train(pairs, TokenizerConfig(vocab_size=8192, special_tokens=special))
tok.save_pretrained("my-tokenizer")      # tokenizer.json, tokenizer_config.json, groups.json

Using a trained tokenizer

Load a tokenizer saved in a local folder:

from grouptok import GroupedTokenizer

tok = GroupedTokenizer.from_pretrained("my-tokenizer")

or one published on the Hugging Face Hub (see Publishing a trained tokenizer). from_pretrained reads local folders only, so download the repository first; huggingface_hub comes with tokenizers:

from huggingface_hub import snapshot_download

tok = GroupedTokenizer.from_pretrained(snapshot_download("your-name/my-tokenizer"))

Then:

ids = tok.encode("The king", bos=True, eos=True)
tok.pairs(ids)                           # [(group, member), ...] for each token
tok.decode(ids)                          # 'The king'
tok.group_of(" king")                    # [' king', ' roi']
tok.readable_groups()[:5]                # groups with at least two members, as token strings

The grouping is a Grouping, a one-to-one map between token ids and (group, member) pairs, which a model needs for its embedding tables and heads:

g = tok.grouping
g.token_group, g.token_member            # per-token group and member ids (tuples of length vocab_size)
g.num_groups, g.max_group_size           # sizes of the group and member tables
g.token(group, member)                   # back to the token id

Publishing a trained tokenizer

Save the trained tokenizer and upload the folder to a repository on the Hugging Face Hub with the hf command. hf upload creates the repository if it doesn't exist yet.

hf auth login                                        # once, with a token that has write access
hf upload your-name/my-tokenizer my-tokenizer        # repository id, then the folder save_pretrained wrote

If hf isn't found, update huggingface_hub: pip install -U huggingface_hub.

Anyone can then download it and load it with grouptok:

from huggingface_hub import snapshot_download
from grouptok import GroupedTokenizer

tok = GroupedTokenizer.from_pretrained(snapshot_download("your-name/my-tokenizer"))

The repository holds a standard tokenizer.json and tokenizer_config.json, so transformers.AutoTokenizer.from_pretrained("your-name/my-tokenizer") also works. It loads the BPE tokenizer only and ignores groups.json.

More than two languages

Pairs don't have to share a language pair: pool en–fr, en–de and fr–de pairs and train once. A token joins the group of whatever it is most often aligned with, so a group can hold " house", " maison" and " Haus".

The group size cap(max_group_size, default 8) bounds how many members a group can get.

from grouptok import GroupedTokenizer, GroupingConfig, TokenizerConfig

pairs = en_fr_pairs + en_de_pairs + fr_de_pairs
tok = GroupedTokenizer.train(pairs, TokenizerConfig(vocab_size=32000, grouping=GroupingConfig(max_group_size=8)))

Sources

Instead of loading the pairs yourself, describe each dataset as a Source, with its columns, row count and, if it needs one, its own aligner, and pass them all to train:

from grouptok import AlignerConfig, GroupedTokenizer, Source, TokenizerConfig

sources = [
    Source("Helsinki-NLP/opus-100", subset="en-fr", columns=("translation.en", "translation.fr"), rows=200_000),
    Source("michsethowusu/english-swahili_sentence-pairs", columns=("English", "Swahili"), rows=200_000),
    Source("Helsinki-NLP/opus-100", subset="en-ha", columns=("translation.en", "translation.ha"),
           aligner=AlignerConfig(model="Davlan/afro-xlmr-large")),
    Source("pairs/en-am.jsonl", columns=("en", "am")),
]
tok = GroupedTokenizer.train(sources, TokenizerConfig(vocab_size=49152, aligner=AlignerConfig(model="FacebookAI/xlm-roberta-base")))

• path is a local .jsonl file, or a Hugging Face dataset id (streamed; needs the datasets extra).
• columns are the two sides of the pairs, in order, and the first one supplies member 0 of the groups it creates (put the language every pair shares first). A dot reads a nested field. Every source names them.
• rows caps the rows read; rows with an empty column are skipped and don't count.
• aligner overrides the training aligner for this source. Pairs are aligned in one pass per aligner, and the link counts of all passes are pooled before grouping. Different encoders link at different rates, so a language aligned by a denser one wins more group slots: prefer one encoder that covers every language, and override only the languages it doesn't cover.

Sources are read into memory before training, like pairs.

Grouping a pretrained tokenizer

Pass a folder holding a pretrained tokenizer's files, tokenizer.json among them, and train groups its vocabulary instead of training a BPE. The vocabulary and the token ids stay the same, so the grouping fits the models that use that tokenizer:

from huggingface_hub import snapshot_download
from grouptok import GroupedTokenizer, TokenizerConfig

qwen = snapshot_download("Qwen/Qwen2.5-0.5B", allow_patterns=["*token*"])   # the tokenizer files only
tok = GroupedTokenizer.train(sources, tokenizer=qwen)
tok.save_pretrained("qwen-grouped")   # Qwen's tokenizer files as they are, plus groups.json

gpt2 = snapshot_download("openai-community/gpt2", allow_patterns=["*token*"])
tok = GroupedTokenizer.train(sources, TokenizerConfig(special_tokens=("<|endoftext|>",) * 3), tokenizer=gpt2)

• Groups can only link tokens that exist. A vocabulary with few whole-word tokens in a language gives that language few groups: multilingual vocabularies (Qwen2.5, Llama 3, Gemma) suit better than English-centric ones like GPT-2's.

Configuration

from grouptok import AlignerConfig, GroupingConfig, TokenizerConfig

TokenizerConfig(
    vocab_size=8192,
    special_tokens=("<|endoftext|>", "<|im_start|>", "<|im_end|>"),   # pad, bos, eos -> ids 0, 1, 2; none by default
    additional_tokens=("<think>", "</think>"),                        # more reserved tokens, never grouped
    chat_template=None,                                               # Jinja template saved to tokenizer_config.json
    grouping=GroupingConfig(max_group_size=8, min_link_count=5, min_dice=0.1),
    aligner=AlignerConfig(model="aneuraz/awesome-align-with-co", layer=8, threshold=1e-3,
                          batch_tokens=8192, max_length=512, device=None, fp16=True),
)

Any Hugging Face encoder with a fast tokenizer can be the aligner (mBERT, XLM-R, awesome-align, ...). You can also pass your own object to GroupedTokenizer.train(..., aligner=...). It needs an encode(texts) method that returns the vectors [batch, length, dim] and the character spans [batch, length, 2] of its word pieces, with empty spans for special and padding positions.

Command line

grouptok train --pairs europarl.jsonl opus.jsonl:src,tgt --columns en,fr --limit 200000 --out my-tokenizer --vocab-size 8192
grouptok inspect my-tokenizer --word " king" " house"
grouptok encode my-tokenizer "The king said"

--pairs takes JSONL files whose two sides are the --columns; write FILE:A,B to choose the fields of one file. grouptok train --help lists every option.

Instead of --pairs, --config takes a JSON list of sources. A source's aligner overrides only the fields it sets; the rest, and the whole aligner of sources without one, come from the aligner options (--aligner, --layer, --device, ...):

grouptok train --config sources.json --out my-tokenizer --vocab-size 49152 --aligner FacebookAI/xlm-roberta-base

[
  {"path": "Helsinki-NLP/opus-100", "subset": "en-fr", "columns": ["translation.en", "translation.fr"], "rows": 200000},
  {"path": "Helsinki-NLP/opus-100", "subset": "en-ha", "columns": ["translation.en", "translation.ha"],
   "aligner": {"model": "Davlan/afro-xlmr-large"}},
  {"path": "pairs/en-am.jsonl", "columns": ["en", "am"]}
]

How it works

1. BPE. A byte-level BPE is trained on both sides of every pair, unless a pretrained tokenizer is given. Reserved tokens come first.

2. Alignment. Each pair goes through the aligner (its source's own, if it has one). A BPE subword gets the mean vector of the aligner's word pieces that overlap it in characters. Two subwords are linked when each one's softmax over the other side's similarities exceeds threshold (awesome-align's rule).

3. Scores. The Dice score of two subwords is 2 * links / (occurrences(a) + occurrences(b)). Identical subwords on the two sides (names, numbers) are not linked.

4. Grouping. Links are visited by descending Dice, skipping pairs below min_link_count or min_dice:

   • if neither subword is grouped, they start a new group;
   • if one is, the other joins its group unless the group already has max_group_size members;
   • two existing groups are never merged.

   Every remaining token, including the reserved tokens and the byte fragments (tokens that are only part of a UTF-8 character), is a group of its own.

Files

save_pretrained writes:

• tokenizer.json: a standard tokenizers file;
• tokenizer_config.json: for transformers.AutoTokenizer;
• groups.json:

{"max_group_size": 8, "token_group": [5120, 5121, 5122, 412, ...], "token_member": [0, 0, 0, 0, ...],
 "special_tokens": ["<|endoftext|>", "<|im_start|>", "<|im_end|>"], "additional_tokens": []}

A pretrained tokenizer, or one loaded with from_pretrained, gets its tokenizer files copied as they are instead of the first two.

token_group and token_member give the (group, member) pair of each token id; max_group_size bounds the member ids. Grouping.load reads these three and ignores any other keys. To read the groups as text, use tok.readable_groups() or grouptok inspect.

Development

Clone the repo

git clone https://github.com/henryhale/grouptok.git
cd grouptok

Install dependencies

pip install torch --index-url https://download.pytorch.org/whl/cpu   # optional: CPU torch is enough for the tests
pip install -e ".[train,test]"
pytest

The tests run offline. They train on a 500-pair English–French sample in tests/data (kept in the repository, not in the package) with a stand-in aligner instead of a downloaded model.

Preview the docs site, built from this README and the docstrings

pip install -e ".[docs]"
mkdocs serve

License

Released under the Apache-2.0 license. See LICENSE for details.

© 2026-present Henry Hale