From a table to a model file

A tokenizer is a vocabulary plus the rules for cutting text and for the tokens a model reserves. This guide builds one from nothing, then reads the three formats a real model ships: a rank table, a SentencePiece .model, and a tokenizer.json. Every block below compiles, and every output was produced by running it.

01Install

moon add moonbitstack/moontoken

Nothing else is needed, and no dependency opens a file. The library parses formats; fetching a vocabulary, caching it or unpacking it from a model archive stays with the caller.

02A table of your own

A byte-level table has to carry every byte value: that is what makes the encoding total over text, and Ranks::from_pairs is where you say so. On top of the 256 bytes go merges — each a byte string and the rank it occupies. The smallest useful table is 256 entries and no merges; this one has seven, which is enough to see the mechanism.

fn ranks() -> @vocab.Ranks raise {
  let pairs : Array[(Bytes, Int32)] = []
  for b = 0; b < 256; b = b + 1 {
    pairs.push((Bytes::make(1, b.to_byte()), b))
  }
  let merges = [
    ("th", 256), ("he", 257), ("the", 258), ("in", 259),
    ("ing", 260), (" and", 261), (" world", 262),
  ]
  for i = 0; i < merges.length(); i = i + 1 {
    let (token, rank) = merges[i]
    pairs.push((@utf8.encode(token), rank))
  }
  @vocab.Ranks::from_pairs(pairs)
}

Then a tokenizer: the published cl100k_base split, and the table you just built. The split is a scanner written in this library, not a regular expression handed to a host runtime, so the same text splits the same way on every backend.

fn tokenizer() -> @moontoken.Tokenizer raise {
  @moontoken.Tokenizer::cl100k(ranks=ranks())
}

fn ids_text(ids : Array[Int32]) -> String {
  let out = StringBuilder()
  let mut gap = ""
  for id in ids {
    out.write_string(gap)
    out.write_string(id.to_string())
    gap = " "
  }
  out.to_string()
}

fn encode_once() -> String raise {
  let tokenizer = tokenizer()
  ids_text(tokenizer.encode("the world and the thing"))
}

258 262 261 32 258 32 256 260

Seven ids for twenty-three bytes. the is one entry, world is one, and is one; the space before the is not merged, so the word and the space are separate. That asymmetry is not a bug in the split — it is what the table was given.

03What a token costs

Counting without encoding allocates nothing you keep: count walks the same split and the same merges, and returns an Int32.

fn counts() -> (Int32, Int32) raise {
  let tokenizer = tokenizer()
  (tokenizer.count("the world and the thing"), tokenizer.length())
}

8  263

The first number is the cost of that sentence; the second is the size of the table, which is what length reports. Both are cheap, and neither touches the network.

04Bytes, and the way back

A byte-level vocabulary can hold entries that are not valid UTF-8 on their own — a stray continuation byte, or the first two bytes of a three-byte character. Decoding them as text would drop them; decode_bytes keeps them, and token_bytes says what any single id stands for.

fn round_trip() -> String raise {
  let tokenizer = tokenizer()
  let ids = tokenizer.encode("the world and the thing")
  let text = tokenizer.decode(ids)
  let raw = tokenizer.decode_bytes(ids)
  let one = match tokenizer.token_bytes(258) {
    Some(b) => b
    None => Bytes::make(0, 0)
  }
  "\\{text} · \\{raw.length()} bytes · id 258 is \\{one.length()} bytes"
}

the world and the thing · 23 bytes · id 258 is 3 bytes

Text out is text in, byte for byte, including input that was never text to begin with.

05A published encoding

OpenAI publishes rank tables as base64, one token rank per line. Ranks::from_tiktoken reads that text; where the text came from is the caller's business.

fn from_published(table : String) -> @moontoken.Tokenizer raise {
  @moontoken.Tokenizer::cl100k(ranks=@vocab.Ranks::from_tiktoken(table))
}

The four published encodings each have a constructor, because each has its own split: Tokenizer::cl100k, ::o200k, ::r50k and ::p50k_edit. When the split is your own, build with Tokenizer::byte_bpe(ranks~, split~, ...) and say which rule to use.

06Reserved tokens

A model reserves spellings such as <|endoftext|> that stand for structure. What happens to text that contains one is a policy, and every policy is explicit:

fn two_readings(tokenizer : @moontoken.Tokenizer) -> (Int32, Int32) raise {
  let text = "<|endoftext|>"
  (
    tokenizer.encode(text, special=@special.Allow).length(),
    tokenizer.encode(text, special=@special.Ignore).length(),
  )
}

1  13

Allowed, the spelling is one token. Ignored, it is the thirteen bytes it is written with. When one reserved spelling is a prefix of another, the longest wins at every position — the library does not take whichever a hash map yields first.

07A SentencePiece model

A .model file is a protobuf that carries everything: the pieces with their scores, the normalisation rules, the trie, the byte fallback, and the symbols the trainer reserved. Tokenizer::of_proto reads it whole; nothing is assumed that the file does not say.

fn from_model(bytes : Bytes) -> @moontoken.Tokenizer raise {
  @moontoken.Tokenizer::of_proto(bytes)
}

Both trainer types are read — unigram, which this library segments by score, and BPE, which merges by rank. A user-defined symbol is taken as one token, and a character the model never saw falls back to its bytes rather than to an unknown id, when the model says so.

08A tokenizer.json

The JSON a model hub ships holds the vocabulary, the merges, the normaliser, the pre-tokenizer and the added tokens in one document. Tokenizer::parse reads it; Tokenizer::of takes the same thing as a Json value you already parsed.

fn from_json(document : String) -> @moontoken.Tokenizer raise {
  @moontoken.Tokenizer::parse(document)
}

Anything the document asks for that this reader does not implement is refused by name. That is the point of the refusal: an approximation would tokenize quietly and wrongly, and the numbers it produced would look fine.

09When it refuses

One error type covers the whole library, and it says where. It lives in moonbitstack/moontoken/refused and is re-exported as @moontoken.Refused; the variants are worth matching on rather than printing, because each carries the piece that went wrong.

fn explain(error : @refused.Refused) -> String {
  match error {
    Malformed(what, at) => "malformed at \\{at}: \\{what}"
    Absent(name) => "absent: \\{name}"
    Unsupported(what) => "unsupported: \\{what}"
    Missing(id) => "no token with id \\{id}"
    Disallowed(what, at) => "disallowed at \\{at}: \\{what}"
  }
}

Missing is the one to expect in production: a stream of ids from a model built on a different table has ids this table does not have, and the number in the error is the first one.

10A model of your own

The Model trait is open: a segmentation this library does not ship can be wrapped in the same tokenizer, keeping the reserved tokens, the policy and the error shape. Five methods are the whole contract.

pub(open) trait Model {
  fn encode(Self, StringView) -> Array[Int32] raise @refused.Refused
  fn decode_bytes(Self, Array[Int32]) -> Bytes raise @refused.Refused
  fn decode(Self, Array[Int32]) -> String raise @refused.Refused
  fn token_bytes(Self, Int32) -> Bytes?
  fn length(Self) -> Int32
}

A tokenizer built with Tokenizer::new(model~, specials~, policy~) then answers encode, count, decode and token_bytes through it. token_bytes returning None is the one answer the tokenizer treats as missing rather than malformed.