sentencepiece

1# SentencePiece - Language-Independent Tokenization
2 
3Unsupervised tokenizer that works on raw text without language-specific preprocessing.
4 
5## When to use SentencePiece
6 
7**Use SentencePiece when:**
8- Building multilingual models (no language-specific rules)
9- Working with CJK languages (Chinese, Japanese, Korean)
10- Need reproducible tokenization (deterministic vocabulary)
11- Want to train on raw text (no pre-tokenization needed)
12- Require lightweight deployment (6MB memory, 50k sentences/sec)
13 
14**Performance**:
15- **Speed**: 50,000 sentences/sec
16- **Memory**: ~6MB for loaded model
17- **Languages**: All (language-independent)
18 
19**Use alternatives instead**:
20- **HuggingFace Tokenizers**: Faster training, more flexibility
21- **tiktoken**: OpenAI models (GPT-3.5/4)
22- **BERT WordPiece**: English-centric tasks
23 
24## Quick start
25 
26### Installation
27 
28```bash
29# Python
30pip install sentencepiece
31 
32# C++ (requires CMake)
33git clone https://github.com/google/sentencepiece.git
34cd sentencepiece
35mkdir build && cd build
36cmake .. && make -j $(nproc)
37sudo make install
38```
39 
40### Train model
41 
42```bash
43# Command-line (BPE with 8000 vocab)
44spm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe
45 
46# Python API
47import sentencepiece as spm
48 
49spm.SentencePieceTrainer.train(
50    input='data.txt',
51    model_prefix='m',
52    vocab_size=8000,
53    model_type='bpe'
54)
55```
56 
57**Training time**: ~1-2 minutes for 100MB corpus
58 
59### Encode and decode
60 
61```python
62import sentencepiece as spm
63 
64# Load model
65…