struct Tokenizer
A tokenizer: a model, the tokens it reserves, and what may happen to them. Text goes in and tokens come out, and the tokens come back as the same text, including the bytes that are not text on their own. The vocabulary is the caller's: this library reads one it is handed and never opens a file.
pub struct Tokenizer {
model : &Model
specials : @special.Table
policy : @special.Policy
}
pub fn Tokenizer::byte_bpe(ranks~ : @vocab.Ranks, split~ : @split.Split, specials? : @special.Table, policy? : @special.Policy, options? : @bpe.Options) -> Self raise @refused.RefusedA tokenizer over a byte-level byte-pair table.
pub fn Tokenizer::cl100k(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.RefusedThe tokens of cl100k_base, given the table of ranks that encoding uses.
pub fn Tokenizer::count(Self, StringView, special? : @special.Policy) -> Int32 raise @refused.RefusedHow many tokens text holds. This is encode with every reserved token recognised, which is the number a caller budgeting a context window wants: the reserved spellings cost their one token rather than being taken apart as text.
pub fn Tokenizer::decode(Self, Array[Int32], reserved? : Bool) -> String raise @refused.RefusedThe text a list of tokens stands for, with U+FFFD standing in for the byte sequences that are not text.
pub fn Tokenizer::decode_bytes(Self, Array[Int32], reserved? : Bool) -> Bytes raise @refused.RefusedThe bytes a list of tokens stands for.
pub fn Tokenizer::encode(Self, StringView, special? : @special.Policy) -> Array[Int32] raise @refused.RefusedThe tokens text holds. special overrides the policy this tokenizer was built with, for a caller that knows this one string came from a model that emits the reserved spelling itself.
pub fn Tokenizer::encode_ordinary(Self, StringView) -> Array[Int32] raise @refused.RefusedThe tokens of text that holds no reserved token, whatever the policy says.
pub fn Tokenizer::length(Self) -> Int32How many tokens the vocabulary has, reserved ones not counted.
pub fn Tokenizer::new(model~ : &Model, specials? : @special.Table, policy? : @special.Policy) -> SelfA tokenizer over any model.
pub fn Tokenizer::o200k(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.RefusedThe tokens of o200k_base, given the table of ranks that encoding uses.
pub fn Tokenizer::o200k_harmony(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.RefusedThe tokens of o200k_harmony, given the table of ranks o200k_base publishes: the same split and the same table, with the control words the harmony chat format writes.
pub fn Tokenizer::of(@json.Json) -> Self raise @refused.RefusedA tokenizer from the tokenizer.json a model distribution ships. The file describes a vocabulary, the merges over it, the normalisation, the cut, and the tokens the model reserves. Everything in it that this reader does not implement is refused by name rather than approximated.
pub fn Tokenizer::of_proto(Bytes) -> Self raise @refused.RefusedA tokenizer from a sentence-piece model file.
pub fn Tokenizer::p50k_edit(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.RefusedThe tokens of p50k_edit, given the table of ranks that encoding uses.
pub fn Tokenizer::parse(StringView) -> Self raise @refused.RefusedA tokenizer from the text of a tokenizer.json file. The text is read as strict RFC 8259: a trailing comma, a comment or a control character inside a string is refused where it is, rather than being quietly accepted and changing what the vocabulary ends up holding.
pub fn Tokenizer::piece_bpe(pieces~ : @piece.Pieces, normalizer? : @normalizer.Normalizer, user_defined? : Array[Bytes], specials? : @special.Table, policy? : @special.Policy) -> SelfA tokenizer over a sentence-piece vocabulary that segments by merging neighbouring symbols.
pub fn Tokenizer::policy(Self) -> @special.PolicyWhat this tokenizer does with the text that holds a reserved token.
pub fn Tokenizer::r50k(ranks~ : @vocab.Ranks, policy? : @special.Policy) -> Self raise @refused.RefusedThe tokens of r50k_base, given the table of ranks that encoding uses. gpt2 and p50k_base are the same split and the same one reserved token over their own tables, so they read through this constructor too.
pub fn Tokenizer::token_bytes(Self, Int32) -> Bytes?The bytes one token stands for.
pub fn Tokenizer::unigram(pieces~ : @piece.Pieces, normalizer? : @normalizer.Normalizer, specials? : @special.Table, policy? : @special.Policy) -> SelfA tokenizer over a sentence-piece vocabulary that segments by the most likely sequence of pieces.
pub fn Tokenizer::with_policy(Self, @special.Policy) -> SelfA tokenizer with a different answer for the text that holds a reserved token.