Quick start

Add the module, import what you use

moon add moonbitstack/moonucd

import {
  "moonbitstack/moonucd",
  "moonbitstack/moonucd/width",
}

fn main {
  println(@moonucd.category('中'))            // Lo
  println(@moonucd.upper("straße"))           // STRASSE
  println(@width.text_width("Hello 世界"))    // 10
}

The root package answers what a character is and how it changes case. The other eight packages each carry their own tables, so a program links only the tables of the packages it imports. Every function is pure and runs on native, wasm, wasm-gc and js.

moonbitstack/moonucd

What a character is

category gives the General_Category, one of 30 two-letter values; unassigned code points are Cn. bidi gives the Bidi_Class, with the defaults the Standard sets for unassigned code points.

@moonucd.category('Dž')          // Lt
@moonucd.category('\u{378}')    // Cn
@moonucd.bidi('א')              // R
@moonucd.bidi('٣')              // AN

numeric tells the three kinds of numeric character apart, and is None for the rest.

has answers 50 binary properties through one enum: identifiers (XidStart, XidContinue, PatternSyntax), case (Uppercase, Cased, ChangesWhenCasefolded), emoji (Emoji, EmojiPresentation, ExtendedPictographic), and the shape of text (WhiteSpace, Dash, QuotationMark, Ideographic, Math). An identifier check is two of them:

fn is_identifier(s : String) -> Bool {
  let mut first = true
  for ch in s {
    let ok = if first {
      @moonucd.has(ch, XidStart)
    } else {
      @moonucd.has(ch, XidContinue)
    }
    if !ok {
      return false
    }
    first = false
  }
  !first
}

is_identifier("变量_1")    // true
is_identifier("naïve")     // true
is_identifier("1st")       // false

moonbitstack/moonucd

Changing case

The mappings are the full ones, so a character can become several, and each function takes and returns a string. lower turns a capital sigma that ends a word into ς.

@moonucd.upper("straße")         // "STRASSE"
@moonucd.upper("ffi")              // "FFI"
@moonucd.lower("ΣΟΦΟΣ ΣΟΦΟΣ")    // "σοφος σοφος"
@moonucd.title("dž")              // "Dž"
@moonucd.fold("Straße")          // "strasse"

title maps every character to its titlecase form; which characters start a word is the caller's choice, and @segment.words finds them. Rules that hold only in one language, such as the Turkish dotless i, are left to the caller as well.

To compare text regardless of case and of how accents are composed, fold between two normalizations:

fn same_text(a : String, b : String) -> Bool {
  let key = fn(s : String) {
    @normalize.normalize(NFC, @moonucd.fold(@normalize.normalize(NFD, s)))
  }
  key(a) == key(b)
}

same_text("Straße", "STRASSE")    // true
same_text("Å", "a\u{30A}")        // true

…/age · …/block · …/name

Age, blocks and names

@age.age('€')                     // Some((2, 1))
@age.age('\u{378}')               // None
@block.block('中')                // Some("CJK Unified Ideographs")
@block.range("Greek")             // Some(('\u{370}', '\u{3FF}'))
@name.name('中')                  // Some("CJK UNIFIED IDEOGRAPH-4E2D")
@name.name('가')                  // Some("HANGUL SYLLABLE GA")
@name.lookup("copyright sign")    // Some('©')
@name.lookup("NULL")              // Some('\u{0}')

age is the (major, minor) version that first assigned the code point. range and lookup match names loosely: case, spaces, underscores and a hyphen between two letters do not count. range also takes a leading "Is" and the short block aliases (ASCII, Greek, CJK); lookup takes formal names and the aliases of NameAliases.txt, and a formal name wins over an alias spelled the same.

…/script

Scripts

@script.script('中')          // Han
@script.script('1')           // Common
@script.script('\u{301}')     // Inherited
@script.extensions('ー')      // [Hiragana, Katakana]
@script.Script::of("latn")    // Some(Latin)

A character used across scripts is Common; a combining mark that takes the script of its base is Inherited. extensions lists the scripts such a character is used with, and is [script(ch)] for the rest. Script::of takes the long name or the four-letter code, in any case.

Mixed scripts in one word are a common sign of a lookalike:

fn scripts_in(s : String) -> Array[@script.Script] {
  let seen = []
  for ch in s {
    match @script.script(ch) {
      Common | Inherited => ()
      sc => if !seen.contains(sc) { seen.push(sc) }
    }
  }
  seen
}

scripts_in("pаypal")    // [Latin, Cyrillic]

…/normalize

Normalization

@normalize.normalize(NFD, "Å")               // "A\u{30A}"
@normalize.normalize(NFC, "A\u{30A}")        // "Å"
@normalize.normalize(NFKC, "ｈｅｌｌｏ")     // "hello"
@normalize.normalize(NFKC, "㍿")             // "株式会社"
@normalize.is_normalized(NFC, "A\u{30A}")    // false
@normalize.combining('\u{301}')              // 230
@normalize.decomposition('½')                // "<fraction> 0031 2044 0032"

NFC is the form to store and compare text in. NFKC also folds compatibility variants (fullwidth letters, ligatures, circled digits, squared words) and suits search keys and identifiers. decomposition gives the field as the database writes it: hex code points, behind a tag such as <compat> or <wide> for a compatibility mapping, and an empty string when there is none. Hangul syllables decompose by rule, so their field is empty too.

…/width

Width

@width.width('A')                              // 1
@width.width('中')                             // 2
@width.width('\u{301}')                        // 0
@width.width('\u{7}')                          // -1
@width.text_width("Hello 世界")                // 10
@width.text_width("👨\u{200D}👩\u{200D}👧")    // 2
@width.east_asian_width('α')                   // A
@width.width('α', ambiguous=2)                 // 2

-1 marks a control character, and a string that holds one has no width. An emoji sequence joined by Zero Width Joiners counts once. East Asian Ambiguous characters (Greek, Cyrillic, box drawing, some punctuation) take one cell; pass ambiguous=2 for a terminal set up for CJK.

Padding by cells instead of by characters keeps columns straight:

fn pad(s : String, cells : Int) -> String {
  s + " ".repeat(cells - @width.text_width(s))
}

…/segment

Segmentation

let s = "🇨🇳e\u{301}👨\u{200D}👩\u{200D}👧!"
s.length()                       // 15 UTF-16 code units
@segment.grapheme_clusters(s)    // ["🇨🇳", "é", "👨‍👩‍👧", "!"]
@segment.words("Mr. Smith's 3.14 cats.")
// ["Mr", ".", " ", "Smith's", " ", "3.14", " ", "cats", "."]
@segment.sentences("Hello. Bye! Fine.")
// ["Hello. ", "Bye! ", "Fine."]

A grapheme cluster is what a reader takes for one character: a flag, a letter with its accents, a family emoji. words keeps the spaces and punctuation between words, so the pieces join back into the text; keep the pieces that hold a letter or a digit to get the words alone. Ideographs come out one per piece.

grapheme_at and word_at take a UTF-16 offset, the index MoonBit strings use, and return the (start, end) of the piece that holds it: one cursor step, one double-click selection.

@segment.grapheme_at("e\u{301}x", 1)    // (0, 2)
@segment.word_at("hello, world", 9)     // (7, 12)

For code that already works on code points, grapheme_boundaries, word_boundaries and sentence_boundaries take an array of them and flag each gap; gcb_property, wb_property and sb_property give the break property of one character.

@segment.grapheme_boundaries([0x65, 0x301, 0x78])    // [false, true]
@segment.wb_property('\'')                           // Single_Quote

…/linebreak

Line breaking

@linebreak.linebreaks("The quick 中文 fox")
// ["The ", "quick ", "中", "文 ", "fox"]
@linebreak.breaks("one two\nthree")    // [(4, false), (8, true)]
@linebreak.lb_property('。')           // CL

linebreaks cuts the text after every break opportunity. breaks gives the same places as UTF-16 offsets, each with whether the line must end there, after a hard line break, or only may. A closing mark such as 。 never starts a line, so it stays with the character before it.

A greedy wrap to a number of terminal cells is a short loop over breaks:

fn wrap(text : String, columns : Int) -> Array[String] {
  let lines = []
  let mut start = 0
  let mut fit = 0
  let line = fn(a, b) { text.view(start_offset=a, end_offset=b).trim_end() }
  for brk in [..@linebreak.breaks(text), (text.length(), true)] {
    let (at, must) = brk
    if fit > start && @width.text_width(line(start, at)) > columns {
      lines.push(line(start, fit).to_owned())
      start = fit
    }
    fit = at
    if must {
      lines.push(line(start, at).to_owned())
      start = at
    }
  }
  lines
}

src/examples/07-terminal runs it, with a truncation that never splits a grapheme cluster:

+------------------------+
|MoonBit 把 Unicode 的断 |
|行规则用在终端里：      |
|English words wrap at   |
|spaces, 中文在字与字之间|
|断开。                  |
+------------------------+

Every package

Versions

@moonucd.version()    // "17.0.0"
@width.version()      // "17.0.0"

Every package reports the Unicode version its tables come from, and they always agree. The tables are generated from the Unicode Character Database by the scripts in scripts/; a new Unicode version is a regeneration, and the interfaces stay as they are.