Skip to content
Navigation Menu
Sign in
Appearance settings
Platform
AI CODE CREATION
GitHub Copilot
Write better code with AI
GitHub Copilot app
Direct agents from issue to merge
MCP Registry
Integrate external tools
DEVELOPER WORKFLOWS
Actions
Automate any workflow
Codespaces
Instant dev environments
Issues
Plan and track work
Code Review
Manage code changes
Code Quality
Enforce quality at merge
APPLICATION SECURITY
GitHub Advanced Security
Find and fix vulnerabilities
Code security
Secure your code as you build
Secret protection
Stop leaks before they start
EXPLORE
Why GitHub
Documentation
Blog
Changelog
Marketplace
View all features
Solutions
BY COMPANY SIZE
Enterprises
Small and medium teams
Startups
Nonprofits
BY USE CASE
App Modernization
DevSecOps
DevOps
CI/CD
View all use cases
BY INDUSTRY
Healthcare
Financial services
Manufacturing
Government
View all industries
View all solutions
Resources
EXPLORE BY TOPIC
AI
Software Development
DevOps
Security
View all topics
EXPLORE BY TYPE
Customer stories
Events & webinars
Ebooks & reports
Business insights
GitHub Skills
SUPPORT & SERVICES
Documentation
Customer support
Community forum
Trust center
Partners
View all resources
Open Source
COMMUNITY
GitHub Sponsors
Fund open source developers
PROGRAMS
Security Lab
Maintainer Community
GitHub Stars
Archive Program
REPOSITORIES
Topics
Trending
Collections
Enterprise
ENTERPRISE SOLUTIONS
Enterprise platform
AI-powered developer platform
AVAILABLE ADD-ONS
GitHub Advanced Security
Enterprise-grade security features
Copilot for Business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Pricing
Search
/
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
hasktorch
/
tokenizers
Public
forked from
huggingface/tokenizers
Notifications
You must be signed in to change notification settings
Fork
5
Star
5
Code
Pull requests
2
Actions
Projects
Security and quality
0
Insights
Additional navigation options
Code
Pull requests
Actions
Projects
Security and quality
Insights
Bump
- #9
#9
Open
tscholak
wants to merge 40 commits into
master
hasktorch/tokenizers:master
from
bump
hasktorch/tokenizers:bump
Copy head branch name to clipboard
Conversation
Commits
40
(40)
Checks
Files changed
Open
Bump
#9
tscholak
wants to merge 40 commits into
master
hasktorch/tokenizers:master
from
bump
hasktorch/tokenizers:bump
Copy head branch name to clipboard
Commits
Commits on May 20, 2021
Add CTC Decoder for Wave2Vec models (#693)
Show description for 2e2e755
3 people
authored
2e2e755
View commit details
Copy full SHA for 2e2e755
Browse repository at this point
Fix SPM conversions (#686)
Show description for 4b0dc6b
LysandreJik
and
n1t0
authored
4b0dc6b
View commit details
Copy full SHA for 4b0dc6b
Browse repository at this point
Bump handlebars from 4.7.6 to 4.7.7 in /bindings/node (#700)
Show description for 3cf957e
dependabot[bot]
authored
3cf957e
View commit details
Copy full SHA for 3cf957e
Browse repository at this point
Bump y18n from 4.0.0 to 4.0.3 in /bindings/node (#708)
Show description for 7574349
dependabot[bot]
authored
7574349
View commit details
Copy full SHA for 7574349
Browse repository at this point
Bump hosted-git-info from 2.8.8 to 2.8.9 in /bindings/node (#702)
Show description for 8f639b4
dependabot[bot]
authored
8f639b4
View commit details
Copy full SHA for 8f639b4
Browse repository at this point
Bump lodash from 4.17.19 to 4.17.21 in /bindings/node (#701)
Show description for bd19584
dependabot[bot]
authored
bd19584
View commit details
Copy full SHA for bd19584
Browse repository at this point
Commits on May 24, 2021
Fix CHANGELOG.md
n1t0
authored
4b7f8c2
View commit details
Copy full SHA for 4b7f8c2
Browse repository at this point
Fix stripping strings containing Unicode characters (#707)
Show description for c046da7
Narsil
and
n1t0
authored
c046da7
View commit details
Copy full SHA for c046da7
Browse repository at this point
Python - prepare for release 0.10.3
n1t0
committed
3a002c1
View commit details
Copy full SHA for 3a002c1
Browse repository at this point
Remove support for Python 3.5 (#714)
Show description for 755e5f5
n1t0
authored
755e5f5
View commit details
Copy full SHA for 755e5f5
Browse repository at this point
Commits on Jul 21, 2021
Fixing tokenizers with 1.53 (updated some dependencies + clippy) (#764)
Narsil
authored
d83772d
View commit details
Copy full SHA for d83772d
Browse repository at this point
Commits on Aug 11, 2021
Clippy 1.54. (#773)
Narsil
authored
256a71c
View commit details
Copy full SHA for 256a71c
Browse repository at this point
Commits on Aug 12, 2021
Bump ws from 7.3.1 to 7.4.6 in /bindings/node (#721)
Show description for 96c122c
dependabot[bot]
authored
96c122c
View commit details
Copy full SHA for 96c122c
Browse repository at this point
Bump glob-parent from 5.1.1 to 5.1.2 in /bindings/node (#734)
Show description for 5d1b0a9
dependabot[bot]
authored
5d1b0a9
View commit details
Copy full SHA for 5d1b0a9
Browse repository at this point
Bump tar from 4.4.13 to 4.4.17 in /bindings/node (#775)
Show description for ab3d3bc
dependabot[bot]
authored
ab3d3bc
View commit details
Copy full SHA for ab3d3bc
Browse repository at this point
Bump path-parse from 1.0.6 to 1.0.7 in /bindings/node (#774)
Show description for 46bed54
dependabot[bot]
authored
46bed54
View commit details
Copy full SHA for 46bed54
Browse repository at this point
Add a way to specify the unknown token in `SentencePieceUnigramTokenizer` python implem (#762)
Show description for da4c7b1
SaulLu
authored
da4c7b1
View commit details
Copy full SHA for da4c7b1
Browse repository at this point
Expand documentation of UnigramTrainer (#770)
Show description for 6616e69
sgugger
and
n1t0
authored
6616e69
View commit details
Copy full SHA for 6616e69
Browse repository at this point
update lexical-core because 0.7.4 doesn't compile (#758)
Show description for 71fb73e
KoichiYasuoka
authored
71fb73e
View commit details
Copy full SHA for 71fb73e
Browse repository at this point
Commits on Aug 13, 2021
Fix typo in documentation (#743)
Show description for c1100dc
kingyiusuen
and
n1t0
authored
c1100dc
View commit details
Copy full SHA for c1100dc
Browse repository at this point
Add SplitDelimiterBehavior to Punctuation constructor (#657)
Show description for e2bf8da
vladdy
authored
e2bf8da
View commit details
Copy full SHA for e2bf8da
Browse repository at this point
Switch git dependencies in Cargo.toml back to regular versions (#728)
Show description for 5982498
geofft
and
n1t0
authored
5982498
View commit details
Copy full SHA for 5982498
Browse repository at this point
Fix word level tokenizer determinism (#718)
Show description for e7dd643
lucacampanella
and
n1t0
authored
e7dd643
View commit details
Copy full SHA for e7dd643
Browse repository at this point
Commits on Aug 31, 2021
Rust - Add from_pretrained on Tokenizer
n1t0
committed
e71e5be
View commit details
Copy full SHA for e71e5be
Browse repository at this point
Python - Add bindings to Tokenizer.from_pretrained
n1t0
committed
e44fdee
View commit details
Copy full SHA for e44fdee
Browse repository at this point
Better export for FromPretrainedParameters
n1t0
committed
6f9e867
View commit details
Copy full SHA for 6f9e867
Browse repository at this point
Node - Add bindings to Tokenizer.from_pretrained
n1t0
committed
528c9a5
View commit details
Copy full SHA for 528c9a5
Browse repository at this point
Update docs for from_pretrained
n1t0
committed
a4d0f3d
View commit details
Copy full SHA for a4d0f3d
Browse repository at this point
Improve READMEs for from_pretrained
n1t0
committed
ad7090a
View commit details
Copy full SHA for ad7090a
Browse repository at this point
Add tests for from_pretrained
n1t0
committed
35c96e5
View commit details
Copy full SHA for 35c96e5
Browse repository at this point
Rust - Prepare for release 0.11.0 (#789)
n1t0
authored
c65b72d
View commit details
Copy full SHA for c65b72d
Browse repository at this point
Commits on Sep 2, 2021
Python - Update Cargo.lock
n1t0
committed
e68aecc
View commit details
Copy full SHA for e68aecc
Browse repository at this point
Bump tar from 4.4.17 to 4.4.19 in /bindings/node (#792)
Show description for 23cf8c6
dependabot[bot]
authored
23cf8c6
View commit details
Copy full SHA for 23cf8c6
Browse repository at this point
Python - Pretty json saving defaults to true (#793)
Show description for b8b584d
n1t0
authored
b8b584d
View commit details
Copy full SHA for b8b584d
Browse repository at this point
Prepare node release (#794)
Show description for 884bfb7
n1t0
and
huan
authored
884bfb7
View commit details
Copy full SHA for 884bfb7
Browse repository at this point
Exclude node 15.x for windows
n1t0
committed
36204c8
View commit details
Copy full SHA for 36204c8
Browse repository at this point
Update esaxx-rs to 0.1.7 to fix building on windows
n1t0
committed
fd316bd
View commit details
Copy full SHA for fd316bd
Browse repository at this point
Commits on Sep 3, 2021
resolve merge conflicts
tscholak
committed
2143a24
View commit details
Copy full SHA for 2143a24
Browse repository at this point
bump
tscholak
committed
bd75835
View commit details
Copy full SHA for bd75835
Browse repository at this point
try to bump cayon-cond
tscholak
committed
e061fe1
View commit details
Copy full SHA for e061fe1
Browse repository at this point
You can’t perform that action at this time.
![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4&size=32){kind=avatar}
