moontoken

The reference

Every public type, function and value, generated from the packages' interfaces and the doc comments in their sources. Grouped by package, alphabetical within a package.

moontoken

The tokenizer: one struct, its methods, and the ways to build one from a table, a SentencePiece model file or a tokenizer.json.

Re-exported

Reachable from this package as well; each type is documented in the package it comes from.

pub using @refused {type Refused}

struct Tokenizer

A tokenizer: a model, the tokens it reserves, and what may happen to them. Text goes in and tokens come out, and the tokens come back as the same text, including the bytes that are not text on their own. The vocabulary is the caller's: this library reads one it is handed and never opens a file.

pub struct Tokenizer {
  model : &Model
  specials : @special.Table
  policy : @special.Policy
}
pub fn Tokenizer::byte_bpe(ranks~ : @vocab.Ranks, split~ : @split.Split, specials? : @special.Table, policy? : @special.Policy, options? : @bpe.Options) -> Self raise @refused.Refused

A tokenizer over a byte-level byte-pair table.

pub fn Tokenizer::cl100k(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.Refused

The tokens of cl100k_base, given the table of ranks that encoding uses.

pub fn Tokenizer::count(Self, StringView, special? : @special.Policy) -> Int32 raise @refused.Refused

How many tokens text holds. This is encode with every reserved token recognised, which is the number a caller budgeting a context window wants: the reserved spellings cost their one token rather than being taken apart as text.

pub fn Tokenizer::decode(Self, Array[Int32], reserved? : Bool) -> String raise @refused.Refused

The text a list of tokens stands for, with U+FFFD standing in for the byte sequences that are not text.

pub fn Tokenizer::decode_bytes(Self, Array[Int32], reserved? : Bool) -> Bytes raise @refused.Refused

The bytes a list of tokens stands for.

pub fn Tokenizer::encode(Self, StringView, special? : @special.Policy) -> Array[Int32] raise @refused.Refused

The tokens text holds. special overrides the policy this tokenizer was built with, for a caller that knows this one string came from a model that emits the reserved spelling itself.

pub fn Tokenizer::encode_ordinary(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of text that holds no reserved token, whatever the policy says.

pub fn Tokenizer::length(Self) -> Int32

How many tokens the vocabulary has, reserved ones not counted.

pub fn Tokenizer::new(model~ : &Model, specials? : @special.Table, policy? : @special.Policy) -> Self

A tokenizer over any model.

pub fn Tokenizer::o200k(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.Refused

The tokens of o200k_base, given the table of ranks that encoding uses.

pub fn Tokenizer::of(@json.Json) -> Self raise @refused.Refused

A tokenizer from the tokenizer.json a model distribution ships. The file describes a vocabulary, the merges over it, the normalisation, the cut, and the tokens the model reserves. Everything in it that this reader does not implement is refused by name rather than approximated.

pub fn Tokenizer::of_proto(Bytes) -> Self raise @refused.Refused

A tokenizer from a sentence-piece model file.

pub fn Tokenizer::p50k_edit(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.Refused

The tokens of p50k_edit, given the table of ranks that encoding uses.

pub fn Tokenizer::parse(StringView) -> Self raise @refused.Refused

A tokenizer from the text of a tokenizer.json file.

pub fn Tokenizer::piece_bpe(pieces~ : @piece.Pieces, normalizer? : @normalizer.Normalizer, user_defined? : Array[Bytes], specials? : @special.Table, policy? : @special.Policy) -> Self

A tokenizer over a sentence-piece vocabulary that segments by merging neighbouring symbols.

pub fn Tokenizer::policy(Self) -> @special.Policy

What this tokenizer does with the text that holds a reserved token.

pub fn Tokenizer::r50k(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.Refused

The tokens of r50k_base, given the table of ranks that encoding uses.

pub fn Tokenizer::token_bytes(Self, Int32) -> Bytes?

The bytes one token stands for.

pub fn Tokenizer::unigram(pieces~ : @piece.Pieces, normalizer? : @normalizer.Normalizer, specials? : @special.Table, policy? : @special.Policy) -> Self

A tokenizer over a sentence-piece vocabulary that segments by the most likely sequence of pieces.

pub fn Tokenizer::with_policy(Self, @special.Policy) -> Self

A tokenizer with a different answer for the text that holds a reserved token.

trait Model

What a tokenizer needs from a segmentation algorithm. The four models this library ships are byte-level byte-pair encoding, the two sentence-piece models, and the pair-merge model of a tokenizer.json file. A caller with a fifth implements this and gets the whole surface: text in, tokens out, and the special tokens around them.

pub(open) trait Model {
  fn encode(Self, StringView) -> Array[Int32] raise @refused.Refused
  fn decode_bytes(Self, Array[Int32]) -> Bytes raise @refused.Refused
  fn decode(Self, Array[Int32]) -> String raise @refused.Refused
  fn token_bytes(Self, Int32) -> Bytes?
  fn length(Self) -> Int32
}
decode

The text a list of tokens stands for.

decode_bytes

The bytes a list of tokens stands for.

encode

The tokens of text that holds no reserved token.

length

How many tokens the vocabulary has.

token_bytes

The bytes one token stands for, when the vocabulary has it.

moontoken/bpe

Byte-pair merging: the byte-level model over a rank table, and the word-level one a tokenizer.json names.

struct ByteBpe

A byte-level byte-pair encoder: text is cut into words, each word is taken as UTF-8 bytes, and the merge table turns those bytes into tokens. Every byte has a token of its own, which is what makes the encoding total: any text, valid or not, has an encoding of at least one token per byte.

pub struct ByteBpe {
  ranks : @vocab.Ranks
  split : @split.Split
  cache : Map[String, Array[Int32]]
  options : Options
}
pub fn ByteBpe::decode(Self, ArrayView[Int32]) -> String raise @refused.Refused

The text a list of tokens stands for, with U+FFFD standing in for the byte sequences that are not text.

pub fn ByteBpe::decode_bytes(Self, ArrayView[Int32]) -> Bytes raise @refused.Refused

The bytes a list of tokens stands for. The result need not be valid UTF-8: a caller that cut a character in half gets the two halves back, which is what makes re-encoding exact.

pub fn ByteBpe::encode(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of text.

pub fn ByteBpe::encode_word(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of one word, without cutting it first.

pub fn ByteBpe::length(Self) -> Int32

How many tokens the vocabulary has.

pub fn ByteBpe::new(ranks~ : @vocab.Ranks, split~ : @split.Split, options? : Options) -> Self raise @refused.Refused

Builds an encoder over a table of ranks. Raises when a single byte has no token, because such a table cannot encode text that happens to contain that byte, and finding out mid-sentence is worse than finding out here.

pub fn ByteBpe::token_bytes(Self, Int32) -> Bytes?

The bytes one token stands for.

enum Decoding

How a token's spelling turns back into text.

pub(all) enum Decoding {
  Alphabet
  Marker
  Plain
}
Alphabet

The spelling is the byte-level alphabet, so every character stands for one byte of the original.

Marker

The spelling uses U+2581 where the text had a space.

Plain

The spelling is the text.

struct Options

Settings that change the work rather than the result.

pub(all) struct Options {
  cache : Bool
  cache_limit : Int32
}
cache

Remember the tokens of the words already seen. A tokenizer sees the same words over and over, so the table pays for itself on the second occurrence, and it never changes a result.

cache_limit

How many words to remember before dropping the whole table. Roughly the size at which a page of text repeats itself.

struct WordBpe

A model that merges neighbouring tokens in the order a list of rules gives. This is the model a tokenizer.json file describes: a vocabulary of spellings, a list of merges, and the order they were learned in. It differs from a byte-level table in that the order is a list rather than a rank, and in that a character the vocabulary lacks is a hole rather than an impossibility.

pub struct WordBpe {
  vocab : Map[String, Int32]
  text_of : Map[Int32, String]
  chars : Map[Char, Int32]
  merges : Map[Int32, Map[Int32, Int32]]
  results : Array[Int32]
  unk_id : Int32
  byte_ids : Array[Int32]
  byte_fallback : Bool
  fuse_unk : Bool
  ignore_merges : Bool
  decoding : Decoding
  normalizer : @normalizer.Normalizer
  split : @split.Split
  alphabet : Array[Char]
  reverse : Map[Int32, Int32]
}
pub fn WordBpe::decode(Self, ArrayView[Int32]) -> String raise @refused.Refused

The text a list of tokens stands for.

pub fn WordBpe::decode_bytes(Self, ArrayView[Int32]) -> Bytes raise @refused.Refused

The bytes a list of tokens stands for.

pub fn WordBpe::encode(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of text.

pub fn WordBpe::encode_word(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of one word, without cutting it first.

pub fn WordBpe::length(Self) -> Int32

How many tokens the vocabulary has.

pub fn WordBpe::new(vocab~ : Array[(String, Int32)], merges~ : Array[(String, String)], options? : WordOptions, decoding? : Decoding, normalizer? : @normalizer.Normalizer, split? : @split.Split) -> Self raise @refused.Refused

Builds a model from a vocabulary and the merges between its tokens. merges are pairs of spellings in the order the trainer learned them; a merge whose sides are not both in the vocabulary is refused, because the result would be a rule that can never fire.

pub fn WordBpe::size(Self) -> Int32

The tokens of text, cut by the pre-tokenizer the file named.

pub fn WordBpe::token_bytes(Self, Int32) -> Bytes?

The spelling of one token.

struct WordOptions

Settings of a word-level byte-pair model.

pub(all) struct WordOptions {
  unk_id : Int32
  byte_fallback : Bool
  ignore_merges : Bool
}
byte_fallback

Whether a character the vocabulary lacks is spelled out as bytes.

ignore_merges

Whether a token chooses the whole word over the merges that would cut it.

unk_id

The id of the token that stands in for a character the vocabulary lacks.

Functions

pub fn byte_alphabet() -> Array[Char]

The alphabet the byte-level version of this model writes its vocabulary in: one character per byte, indexed by the byte. The bytes that are printable in Latin-1 keep their own character and the rest are laid out over the code points above U+0100, so that a token never holds a control character or half of a character.

Values

pub let options : Options

The settings a tokenizer starts with: the cache on, at 16384 words.

pub let word_options : WordOptions

The settings a word-level model starts with: no unknown id, no byte fallback, and merges taken even when a whole word is in the vocabulary.

moontoken/normalizer

The rule table a SentencePiece model carries, and the operations a tokenizer.json lists.

enum Mode

Which of the three normalisations this is.

pub enum Mode {
  Identity
  Rule
  Chain
}

struct Normalizer

The normalisation applied before segmentation. A model file carries its own rule table, which is why this is not just a list of operations: the table is a trie of a few thousand replacements, and applying it is what makes two spellings of one character segment alike.

pub struct Normalizer {
  mode : Mode
  add_dummy_prefix : Bool
  remove_extra_whitespaces : Bool
  escape_whitespaces : Bool
  treat_whitespace_as_suffix : Bool
  trie : Array[UInt32]
  replacements : Bytes
  user_defined : Map[Int32, Array[Bytes]]
  stages : Array[Stage]
}
pub fn Normalizer::apply(Self, StringView) -> String raise @refused.Refused

The text as it goes into segmentation.

pub fn Normalizer::chain(Array[Stage]) -> Self

A normalisation from a list of operations, applied in order.

pub fn Normalizer::identity() -> Self

A normalisation that changes nothing.

pub fn Normalizer::rule(spec~ : Spec, user_defined? : Array[Bytes], treat_whitespace_as_suffix? : Bool) -> Self raise @refused.Refused

The rule table of a model file. user_defined are the symbols the caller reserved: the normaliser hands them through untouched, so that a piece spelled the way the caller wrote it stays findable.

pub fn Normalizer::strips_every_leading_marker(Self) -> Bool

Whether every marker at the front comes off, not only the first.

pub fn Normalizer::strips_leading_marker(Self) -> Bool

Whether a word-break marker at the very front of the text stands for a space the text does not have, rather than one it does. A model that adds a leading space puts a marker there on the way in, and taking it off again on the way out is what makes the round trip exact. A model that also drops runs of whitespace keeps taking them off until the text has a character in it, because everything it saw at the front was whitespace it meant to discard.

struct Spec

The normalisation a model expects: the rule table it was trained with, and the three switches around whitespace.

pub(all) struct Spec {
  name : String
  charsmap : Bytes
  add_dummy_prefix : Bool
  remove_extra_whitespaces : Bool
  escape_whitespaces : Bool
}

enum Stage

One step of a normalisation that is a list of Unicode operations rather than a rule table.

pub(all) enum Stage {
  Nfc
  Nfd
  Nfkc
  Nfkd
  Lowercase
  Prepend(String)
  Pad(String)
  Replace(String, String)
  Trim
  EscapeSpace
}
EscapeSpace

Replace a space with U+2581, the marker pieces use for a word break.

Lowercase

Unicode lower case.

Nfc

Canonical composition.

Nfd

Canonical decomposition.

Nfkc

Compatibility composition.

Nfkd

Compatibility decomposition.

Pad

Put this text in front unless the text already starts with it, which is how a cut that keeps its word break stays honest about a text that has one already.

Prepend

Put this text in front.

Replace

Replace every occurrence of the first text with the second.

Trim

Drop whitespace from both ends.

Functions

pub fn longest_reserved(Map[Int32, Array[Bytes]], Bytes, Int32) -> Int32?

The number of bytes of the longest reserved symbol that starts at at.

pub fn user_index(Array[Bytes]) -> Map[Int32, Array[Bytes]]

An index of the symbols a caller reserved, by their first byte, so that finding the one at a position costs a map lookup rather than a scan of the whole list.

moontoken/piece

The SentencePiece model file: its pieces, both trainer types, and the trie it compiles to.

enum Kind

What a piece is for.

pub(all) enum Kind {
  Normal
  Unknown
  Control
  UserDefined
  Unused
  RawByte
}
Control

A marker such as <s>: never produced from text, never decoded.

Normal

segmentation, whether it survives decoding, and whether it stands for a byte. An ordinary piece.

RawByte

One byte, written <0x41>, used when a character has no piece.

Unknown

The piece that stands in for text nothing else covers.

Unused

A piece kept in the vocabulary but never produced.

UserDefined

A piece the caller declared, which is matched before any other rule.

enum ModelKind

What kind of model a file was trained as.

pub(all) enum ModelKind {
  Unigram
  Bpe
  Word
  Char
}
Bpe

Segmentation by merged pairs, the sentence-piece flavour.

Char

One piece per character.

Unigram

Segmentation by the most likely sequence of pieces.

Word

One piece per word.

struct Piece

One entry of a vocabulary: the text it stands for, the log probability the trainer gave it, and what it is.

pub(all) struct Piece {
  text : Bytes
  score : Float
  kind : Kind
}

struct PieceBpe

A model that segments text by merging neighbouring symbols, the sentence-piece flavour: symbols are merged in order of the score of the piece they would form, until no merge is left that the vocabulary has.

pub struct PieceBpe {
  pieces : Pieces
  normalizer : @normalizer.Normalizer
  user_defined : Map[Int32, Array[Bytes]]
}
pub fn PieceBpe::decode(Self, ArrayView[Int32]) -> String raise @refused.Refused

The text a list of tokens stands for.

pub fn PieceBpe::encode(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of text.

pub fn PieceBpe::length(Self) -> Int32

How many pieces the vocabulary has.

pub fn PieceBpe::new(pieces~ : Pieces, normalizer? : @normalizer.Normalizer, user_defined? : Array[Bytes]) -> Self

A BPE SentencePiece model over a vocabulary: pieces in vocabulary order, normalizer for the text before it is cut, and the user-defined symbols that the file reserves.

pub fn PieceBpe::pieces(Self) -> Pieces

The vocabulary.

pub fn PieceBpe::token_bytes(Self, Int32) -> Bytes?

The token a piece id stands for.

struct Pieces

A vocabulary of pieces, with the indexes a model needs: one for the pieces segmentation may use, one for everything.

pub struct Pieces {
  items : Array[Piece]
  exact : Map[Bytes, Int32]
  lattice : Map[Bytes, Int32]
  longest : Int32
  unk_id : Int32
  unk_surface : Bytes
  byte_ids : Array[Int32]
  byte_fallback : Bool
}
pub fn Pieces::at(Self, Int32) -> Piece raise @refused.Refused

The piece with this id.

pub fn Pieces::byte_fallback(Self) -> Bool

Whether a character with no piece may be spelled out as bytes.

pub fn Pieces::byte_id(Self, Int32) -> Int32

The id of the piece for one byte, or -1.

pub fn Pieces::id(Self, BytesView) -> Int32?

The id of a piece, whatever kind it is.

pub fn Pieces::lattice_id(Self, BytesView) -> Int32?

The id of a piece segmentation may use: an ordinary one or a declared one.

pub fn Pieces::length(Self) -> Int32

How many pieces there are; ids run from 0 to this less one.

pub fn Pieces::longest(Self) -> Int32

The longest piece segmentation may use, in bytes.

pub fn Pieces::new(items~ : Array[Piece], unk_id? : Int32, unk_surface? : String, byte_fallback? : Bool) -> Self raise @refused.Refused

Builds the tables over a list of pieces. unk_id is the piece that stands in for text nothing covers, and byte_fallback says whether the model can spell a character out as bytes.

pub fn Pieces::unk_id(Self) -> Int32

The id of the piece that stands in for uncovered text, or -1.

pub fn Pieces::unk_surface(Self) -> Bytes

The text an unknown token stands for when it is decoded.

struct Proto

A model file: its pieces, how it was trained, and how text is normalised before segmentation.

pub struct Proto {
  pieces : Pieces
  trainer : Trainer
  normalizer : @normalizer.Spec
}
pub fn Proto::normalizer(Self) -> @normalizer.Normalizer raise @refused.Refused

The normalisation the model expects, ready to apply.

pub fn Proto::of(Bytes) -> Self raise @refused.Refused

Reads a model file.

pub fn Proto::pieces(Self) -> Pieces

The vocabulary.

pub fn Proto::spec(Self) -> @normalizer.Spec

The normalisation the model expects, as written in the file.

pub fn Proto::trainer(Self) -> Trainer

How the model was trained.

pub fn Proto::user_defined(Self) -> Array[Bytes]

The symbols the caller reserved, in the order the file lists them.

struct Span

What segmentation produced, before the tokens a model reserves are dealt with: a span of the normalised bytes and the piece that covers it.

pub struct Span {
  start : Int32
  stop : Int32
  id : Int32
}

struct Trainer

The settings a model was trained with that change how it runs.

pub(all) struct Trainer {
  kind : ModelKind
  unk_id : Int32
  byte_fallback : Bool
  treat_whitespace_as_suffix : Bool
  unk_piece : String
  unk_surface : String
}

struct Unigram

A model that segments text by the most likely sequence of pieces. Each piece carries the log probability the trainer measured for it, and the segmentation is the one whose pieces add up to the largest total. Text that no piece covers becomes the unknown piece, or, when the model can spell it out, one token per byte.

pub struct Unigram {
  pieces : Pieces
  normalizer : @normalizer.Normalizer
  min_score : Float
}
pub fn Unigram::decode(Self, ArrayView[Int32]) -> String raise @refused.Refused

The text a list of tokens stands for.

pub fn Unigram::encode(Self, StringView) -> Array[Int32] raise @refused.Refused

The tokens of text.

pub fn Unigram::length(Self) -> Int32

How many pieces the vocabulary has.

pub fn Unigram::new(pieces~ : Pieces, normalizer? : @normalizer.Normalizer) -> Self

Builds the model over a vocabulary. Raises when the vocabulary has neither an unknown piece nor byte fallback, because text outside the vocabulary would then have no encoding at all.

pub fn Unigram::pieces(Self) -> Pieces

The vocabulary.

pub fn Unigram::token_bytes(Self, Int32) -> Bytes?

The token a piece id stands for, for callers that want the spelling.

Functions

pub fn byte_of_piece(Bytes) -> Int32?

The byte a <0xNN> spelling stands for, when it is one.

pub fn byte_piece(Int32) -> String

The text of the piece that stands for one byte.

moontoken/refused

The one error shape the library raises, with the offset, the name or the id that went wrong.

suberror Refused

Why an input was refused. Malformed carries the offset, Absent the name of what was missing, and Missing the id that is not in the vocabulary, so that a caller reading a table of a hundred thousand entries can find the one that broke.

pub(all) suberror Refused {
  Malformed(String, Int32)
  Absent(String)
  Unsupported(String)
  Missing(Int32)
  Disallowed(String, Int32)
}
Absent

A required piece of the input is not there.

Disallowed

The text holds a token that the caller did not allow here.

Malformed

The bytes are not the shape this reader accepts.

Missing

There is no token with this id.

Unsupported

The input asks for something this reader does not implement.

moontoken/special

The tokens a model reserves, the table that holds them, and the policy for text that contains one.

struct Hit

Where a special token was found.

pub(all) struct Hit {
  start : Int32
  text : String
  id : Int32
}
id

The id it stands for.

start

The offset of its first code unit.

text

The text it is written as.

enum Policy

What may happen to the tokens a model reserves. A model reserves a handful of spellings, <|endoftext|> and the like, that stand for structure rather than text. Whether text that happens to contain one is taken as the token, taken as text, or refused is the caller's decision, never the library's.

pub(all) enum Policy {
  Refuse
  Allow
  Ignore
  Only(Array[String])
}
Allow

Every reserved token in the table is recognised.

Ignore

No reserved token is recognised; text that holds one is encoded as text.

Only

Only the named tokens are recognised, and the rest are refused.

Refuse

No reserved token is recognised, and text that holds one is refused.

pub fn Policy::allows(Self, StringView) -> Bool

Whether policy recognises text.

pub fn Policy::recognizes(Self) -> Bool

Whether the policy looks for reserved tokens at all.

struct Table

The tokens a model reserves, by spelling.

pub struct Table {
  texts : Array[String]
  ids : Array[Int32]
  by_first : Map[Int32, Array[Int32]]
  index_of : Map[String, Int32]
  text_of : Map[Int32, String]
}
pub fn Table::add(Self, String, Int32) -> Unit

Adds a spelling. An empty spelling is ignored: it would match everywhere.

pub fn Table::empty() -> Self

A table with nothing in it.

pub fn Table::find(Self, StringView, Int32) -> Hit?

The longest reserved token at or after from, and where it stands.

pub fn Table::id(Self, StringView) -> Int32?

The id of a spelling.

pub fn Table::length(Self) -> Int32

How many tokens the table holds.

pub fn Table::of(Array[(String, Int32)]) -> Self

A table from spellings and their ids. A spelling that appears twice keeps the first id, which is how a model file that lists the same control token twice still has one answer.

pub fn Table::text(Self, Int32) -> String?

The spelling of an id, when the id is one the table holds.

Functions

pub fn cl100k_base() -> Table

The special tokens of cl100k_base, and of o200k_base and the two earlier encodings, which share the first one. The ids are the ones the published encodings give them, and the file of mergeable ranks does not carry them: they are part of the encoding's definition, not of the table a caller loads.

pub fn o200k_base() -> Table

The special tokens of o200k_base.

pub fn p50k_edit() -> Table

The special tokens of p50k_edit, the pair of r50k_base plus the three fill-in-the-middle markers.

pub fn r50k_base() -> Table

The special token of r50k_base and p50k_base.

moontoken/split

The pre-tokenizers of the published encodings, written as scanners, and the split a caller can compose.

struct Digits

Whether digits are cut one at a time or as a run.

pub(all) struct Digits {
  individual : Bool
}

enum Split

How text is cut into the words a model is applied to. The three named rules are the published pre-tokenizers of the byte-level encodings: r50k/p50k/gpt2, cl100k_base and o200k_base. They are scanners here rather than regular expressions because the patterns lean on lookahead and possessive quantifiers, and because a tokenizer runs them once per word over the whole input.

pub(all) enum Split {
  Whole
  Gpt2
  Cl100k
  O200k
  Whitespace
  Metaspace
  Digits(Digits)
  Sequence(Array[Split])
}
Cl100k

'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s

Digits

Digits on their own, either one by one or as runs.

Gpt2

'(?:[sdmt]|ll|ve|re)| ?\p{L}++| ?\p{N}++| ?[^\s\p{L}\p{N}]++|\s++$|\s+(?!\S)|\s

Metaspace

A word starts at every U+2581, the marker that whitespace becomes during normalisation. Text that arrived unescaped keeps its spaces inside words.

O200k

The o200k_base rule, which separates runs of capitals from runs of lower-case letters, so that word and Word are different words.

Sequence

The rules applied one after another.

Whitespace

Runs of non-whitespace; the whitespace between them is dropped.

Whole

The whole text is one word.

pub fn Split::pieces(Self, StringView) -> Array[StringView]

The words of text, in order. Joining them gives back the text under every rule but Whitespace, which drops what it splits on.

Functions

pub fn utf8_width_at(Bytes, Int32) -> Int32

The width of the UTF-8 sequence that starts at at, or 0 when the bytes there are not one.

Values

pub let digits_individual : Digits

Digits cut one at a time, which is what the GPT-2 family does and what makes a number cost one token per figure.

pub let digits_run : Digits

Digits kept in a run, so a number costs one token until it meets a merge.

moontoken/vocab

A rank table: from the base64 text a model publishes, from pairs, or empty and filled by hand.

struct Ranks

The mergeable ranks of a byte-level encoding: byte sequences and the order in which the trainer picked them. A rank doubles as the token id, which is how the published encodings are numbered, and the reason a table loaded here needs no second index.

pub struct Ranks {
  encoder : Map[Bytes, Int32]
  decoder : Map[Int32, Bytes]
}
pub fn Ranks::empty() -> Self

An empty table.

pub fn Ranks::from_pairs(Array[(Bytes, Int32)]) -> Self raise @refused.Refused

A table from tokens and their ranks. Raises when one token or one rank appears twice: a table like that has no single answer for an id, and the encodings never contain one.

pub fn Ranks::from_tiktoken(StringView) -> Self raise @refused.Refused

Reads the layout the published encodings ship: one base64 rank pair per line, in whatever order.

pub fn Ranks::get(Self, BytesView) -> Int32?

The rank of a token, or None when the table does not have it.

pub fn Ranks::length(Self) -> Int32

How many tokens the table holds.

pub fn Ranks::missing_bytes(Self) -> Array[Int32]

The bytes that have no token of their own. A byte-level encoding is total over byte strings only when every byte is in the table; without this check an encoding built from a partial table would fail in the middle of a word instead of at the door.

pub fn Ranks::token(Self, Int32) -> Bytes?

The token of a rank.