Text into tokens,
and back.
A tokenizer is three things: a way to cut text into the words a model was trained on, a vocabulary that says what each of those words costs, and a rule for the tokens a model reserves. This library implements all three for the two families in use today, and takes the vocabulary as an argument.
- tokens
- —
- bytes
- —
- bytes / token
- —
The table is the argument
Where a table comes from is the format's business, not the tokenizer's: a GGUF file, a download from a model hub, a string in a repository. This library parses the formats and takes the bytes; fetching them is the caller's. The table below is the one the stick above runs on — — entries, every byte value included, because a table with a gap is not total over text.
A merge is what makes a word cost less than its letters. The merges in this table were written by hand to show the mechanism; a published encoding has a hundred thousand, trained on a corpus. The shape of the thing does not change: an id, the bytes it stands for, and the order in which pairs were fused.
Text this table has never merged falls back to one token per byte. That is not a failure of the reader — it is the reason a vocabulary is worth training: an unfamiliar script costs three tokens a character, and the table above says so plainly.
Three ways to bring one
Byte-level BPE
The encodings that ship with published models, each with its own pre-tokenizer. The split is a scanner, not a regular expression run through a host library, so it behaves the same on every backend.
Tokenizer::cl100kTokenizer::o200kTokenizer::r50kTokenizer::p50k_editTokenizer::byte_bpe
SentencePiece
A .model file carries the trie, the normalisation rules, the byte fallback and the symbols its trainer reserved. Both trainer types are read, and nothing is assumed that the file does not say.
Tokenizer::of_proto- unigram and BPE
- user-defined symbols
tokenizer.json
The JSON a model hub ships: vocabulary, merges, normaliser, pre-tokenizer and added tokens in one document. What the document asks for and this reader does not implement is refused by name, never approximated.
Tokenizer::parse- BPE and unigram
Tokenizer::of
Reserved tokens are the caller's decision
A model reserves a handful of spellings that stand for structure rather than text. Whether text that happens to contain <|endoftext|> is taken as the token, taken as text, or refused belongs to the caller, never to the library.
| Policy | What happens to a reserved spelling |
|---|---|
| Refuse | The text is refused, with the offset of the spelling. |
| Allow | Every reserved spelling is taken as its token. |
| Ignore | Reserved spellings are encoded as ordinary text. |
| Only(names) | Just the spellings named are taken as tokens; the rest are text. |
When one spelling is a prefix of another the longest wins, at every position. A tokenizer cannot define it any other way, and the test that pins the difference is named for it.
fn only_reserved(tokenizer : @moontoken.Tokenizer, text : String) -> Array[Int32] raise {
tokenizer.encode(text, special=@special.Only(["<|endoftext|>"]))
}
What is in the box
| Package | What it is |
|---|---|
| moontoken | The tokenizer: encode, encode_ordinary, decode, count, and the ways to build one. |
| split | The pre-tokenizers of the published encodings, written as scanners. |
| vocab | A rank table, from the base64 file a model publishes or from pairs. |
| bpe | Byte-pair merging: the byte-level model and the word-level one a tokenizer.json names. |
| piece | The SentencePiece model file: both trainer types, and the trie it compiles. |
| normalizer | The rule table a model file carries, and the operations a tokenizer.json lists. |
| special | The tokens a model reserves, and what may happen to them. |
| refused | The one error shape the whole library raises. |