fix(bpe): drop heap entries with non-positive pair_counts before u64 cast - #2389

fix(bpe): drop heap entries with non-positive pair_counts before u64 cast#2389

okxint wants to merge 1 commit into

huggingface:mainhuggingface/tokenizers:mainfrom

okxint:fix/bpe-negative-pair-count-phantom-mergeokxint/tokenizers:fix/bpe-negative-pair-count-phantom-mergeCopy head branch name to clipboard