Skip to content

bpe: do not slice merge tokens at a byte offset that may not be a boundary - #2398

Open
apollo-2006 wants to merge 1 commit into
huggingface:mainfrom
apollo-2006:fix-bpe-merge-prefix-slicing
Open

bpe: do not slice merge tokens at a byte offset that may not be a boundary#2398
apollo-2006 wants to merge 1 commit into
huggingface:mainfrom
apollo-2006:fix-bpe-merge-prefix-slicing

bpe: do not slice merge tokens at a byte offset that may not be a bou…

8797c83
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs