Release 0.23.1 introduced a performance enhancement:
Truncation early exit (#1990)
Right-direction truncation no longer pre-tokenizes past max_length. The new truncation_benchmark doesn't exist on v0.22.2 so there's no apples-to-apples here, but the PR's own measurements on the same machine showed −20–28% across a range of max_length values for right-truncation; left-truncation unchanged.
This change alters the return value of overflowing in the Encoding:
from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers.pre_tokenizers import Whitespace
vocab = {"[UNK]": 0, **{str(i): i + 1 for i in range(100)}}
tok = Tokenizer(WordLevel(vocab=vocab, unk_token="[UNK]"))
tok.pre_tokenizer = Whitespace()
tok.enable_truncation(max_length=10, stride=2)
enc = tok.encode(" ".join(str(i) for i in range(50)))
print(len(enc.overflowing)) # in v0.22.2 this is length 5. in v0.23.1, it is length 0.We use the truncation feature and overflow attribute to conveniently tokenize full sequences. The new release results in a length-0 enc.overflowing even when the input text is of sufficient length to trigger overflow.
The docs still describe Encoding.overflowing as the place to find this data:
A List of overflowing Encoding
When using truncation, the Tokenizer takes care of splitting the output into as many pieces as required to match the specified maximum length. This field lets you retrieve all the subsequent pieces.
At a minimum I think this should be documented as a breaking change in the release notes, rather than somewhat hidden under performance enhancements. However, if I may, I also wonder whether the behaviour should be reverted to retain Encoding.overflow as-is, and a user provided input argument is needed to opt in to the performance enhancement. I haven't looked at the implementation so I don't know how feasible that is.
Release 0.23.1 introduced a performance enhancement:
This change alters the return value of
overflowingin the Encoding:We use the truncation feature and overflow attribute to conveniently tokenize full sequences. The new release results in a length-0
enc.overflowingeven when the input text is of sufficient length to trigger overflow.The docs still describe Encoding.overflowing as the place to find this data:
At a minimum I think this should be documented as a breaking change in the release notes, rather than somewhat hidden under performance enhancements. However, if I may, I also wonder whether the behaviour should be reverted to retain Encoding.overflow as-is, and a user provided input argument is needed to opt in to the performance enhancement. I haven't looked at the implementation so I don't know how feasible that is.