huggingface-tokenizers

1# HuggingFace Tokenizers - Fast Tokenization for NLP
2 
3Fast, production-ready tokenizers with Rust performance and Python ease-of-use.
4 
5## When to use HuggingFace Tokenizers
6 
7**Use HuggingFace Tokenizers when:**
8- Need extremely fast tokenization (<20s per GB of text)
9- Training custom tokenizers from scratch
10- Want alignment tracking (token → original text position)
11- Building production NLP pipelines
12- Need to tokenize large corpora efficiently
13 
14**Performance**:
15- **Speed**: <20 seconds to tokenize 1GB on CPU
16- **Implementation**: Rust core with Python/Node.js bindings
17- **Efficiency**: 10-100× faster than pure Python implementations
18 
19**Use alternatives instead**:
20- **SentencePiece**: Language-independent, used by T5/ALBERT
21- **tiktoken**: OpenAI's BPE tokenizer for GPT models
22- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)
23 
24## Quick start
25 
26### Installation
27 
28```bash
29# Install tokenizers
30pip install tokenizers
31 
32# With transformers integration
33pip install tokenizers transformers
34```
35 
36### Load pretrained tokenizer
37 
38```python
39from tokenizers import Tokenizer
40 
41# Load from HuggingFace Hub
42tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
43 
44# Encode text
45output = tokenizer.encode("Hello, how are you?")
46print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']
47print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]
48 
49# Decode back
50text = tokenizer.decode(output.ids)
51print(text)  # "hello, how are you?"
52```
53…