Guide — moonast

Step 1

Install

Add moonast to the module that will hold your generator.

moon add moonbitstack/moonast

Four packages come with it. A package that only builds code imports the first three:

// moon.pkg
import {
  "moonbitstack/moonast/quote",
  "moonbitstack/moonast/attr",
  "moonbitstack/moonast/gen",
}

quote, attr and gen build for every target. cli reads and writes files, so it builds for native and wasm — which is what a generator binary runs on. Such a binary needs all four.

Step 2

Your first template

A template is a piece of MoonBit with $holes in it. Fill the holes and render:

let source =
  #|fn $name() -> $T {
  #|  $value
  #|}
let code = @quote.items(source)
  .set("name", @quote.ident("answer"))
  .set("T", @quote.ident("Int"))
  .set("value", @quote.int(42))
  .render()
println(code)

What comes out is a whole declaration, laid out the way moon fmt lays it out:

output

///|
fn answer() -> Int {
  42
}

Each constructor decides what a hole may stand for:

Call	Makes	Fills
`@quote.ident(n)`	a name, optionally `@pkg.name`	a value, a field, a type, a constructor, a binder
`@quote.string(s)`	a string literal	an expression, or text inside a string literal
`@quote.int(n)`, `@quote.bool(b)`	a literal	an expression
`@quote.expr(s)`, `@quote.ty(s)`	a parsed expression or type	an expression or a type
`@quote.items(s)`	top-level declarations	the whole file

A template is parsed, not concatenated. If it is not MoonBit once its holes are in place, render refuses with the position of the problem instead of producing something that will not compile.

Step 3

What holes take

The case of a hole's name is part of the grammar: a hole where a type or a constructor goes starts upper case, one where a value or a binder goes starts lower case.

Hole	Takes	Written as	Gives
`$T`	a type or a constructor	`fn f(x : $T)`	`fn f(x : Int)`
`$name`	a value, an expression, or a name	`$value + 1`	`41 + 1`
`@pkg.$name`	a name inside a package	`app : @web.$T`	`app : @web.App`
`$name*`	a list, spliced among statements, array elements or arguments	`[$items*]`	`[1, 2, 3]`
`$name` in a string	a name or a string, as text	`"field $key"`	`"field first name"`
`$$`	a literal dollar sign	`"$$5"`	`"$5"`

A list hole is filled with splice, and it expands where it stands:

println(
  @quote.expr("[$items*]")
  .splice("items", [@quote.int(1), @quote.int(2), @quote.int(3)])
  .render(),
)

output

[1, 2, 3]

The same list hole works among statements:

let source =
  #|fn main {
  #|  $body*
  #|}
println(
  @quote.items(source)
  .splice("body", [
    @quote.expr("println(1)").render(),
    @quote.expr("println(2)").render(),
  ])
  .render(),
)

output

///|
fn main {
  println(1)
  println(2)
}

A list hole among the arguments of a method call, r.add($a, $tags*), is not spliced — it becomes one block expression. Give it an array literal, r.add($a, [$tags*]), or call a plain function, add(r, $a, $tags*).

Two more: a qualified name, and the dollar sign itself.

println(@quote.ty("@web.$T").set("T", @quote.ident("App")).render())
println(@quote.expr("\"$$5\"").render())

output

@web.App
"$5"

Text goes through the official escaping, so a string with Chinese in it comes out as a correct MoonBit literal:

println(
  @quote.items("fn label() -> String { $text }")
  .set("text", @quote.string("坐标：x 与 y"))
  .render(),
)

output

///|
fn label() -> String {
  "坐标：x 与 y"
}

Step 4

Read a declaration

@gen.read takes a file name and its source and gives the declarations as plain views, in source order.

let source =
  #|#demo.describe
  #|pub struct Point {
  #|  x : Int
  #|  y : Int
  #|}
  #|
  #|pub enum Color {
  #|  Red
  #|  Green
  #|  Blue
  #|}
  #|
  #|pub fn dist(a : Point, b : Point, scale~ : Double) -> Double {
  #|  0.0
  #|}
for decl in @gen.read("models.mbt", source) {
  match decl {
    Struct(name~, fields~, ..) =>
      println("struct " + name + " " + fields.map(f => f.name + ": " + f.ty).join(", "))
    Enum(name~, constructors~, ..) =>
      println("enum " + name + " " + constructors.join(" | "))
    Func(owner~, name~, params~, returns~, ..) =>
      println(
        "func " + owner.unwrap_or("-") + "::" + name + " " +
        params.map(p => p.name.unwrap_or("_")).join(", ") +
        " -> " + returns.unwrap_or("Unit"),
      )
    Other(..) => println("other")
  }
}

output

struct Point x: Int, y: Int
enum Color Red | Green | Blue
func -::dist a, b, scale -> Double

Types are kept as the source text that wrote them, which is exactly what you splice into a template. Every view also keeps the annotations that were on the declaration, the file it came from, and the official syntax tree in raw for anything the view leaves out.

A method declared as fn Point::norm arrives as a Func whose owner is Point; a method written inside an impl block arrives as Other, with the block in raw. Decl::attrs() answers the annotations of any of them, and none for Other.

for decl in @gen.read("p.mbt", "fn Point::norm(self : Point) -> Double {\n  0.0\n}") {
  if decl is Func(owner~, name~, ..) {
    println(owner.unwrap_or("-") + "::" + name)
  }
}

output

Point::norm

Step 5

Annotation arguments

An annotation is found by its full name, lib.name for #lib.name(...):

let source =
  #|#app.get("/users", summary="List users", tags="users", tags="public")
  #|fn list_users() -> Array[String] {
  #|  []
  #|}
guard @gen.read("routes.mbt", source) is [Func(name~, attrs~, ..)] else {
  abort("the source declares one function")
}
guard @attr.find(attrs, "app.get") is Some(annotation) else {
  abort("the function is annotated")
}
println(name)

output

list_users

Three questions cover most generators: find for the first one, all for every one of that name, under for everything under a prefix — a generator for #app.get, #app.post and the rest asks once for app.

Step 6

Give the annotation a spec

A Spec says what the annotation takes. Build it by chaining, then bind:

let spec = @attr.Spec::new("app.get")
  .positional("path", Str)
  .optional("summary", Str)
  .many("tags", Str)
let args = spec.bind(annotation)
println(args.str("path").unwrap_or("/"))
println(args.strs("tags").join(", "))
println(args.has("page"))

output

/users
users, public
false

Builder	Means
`positional(key, kind)`	required, given by position in the order these are declared, or by name
`required(key, kind)`	required, given by name
`optional(key, kind)`	may be left out
`many(key, kind)`	may be given any number of times — `tags="a", tags="b"`

A kind is Str, Bool, Ident or Type. Type is a string holding a MoonBit type, checked by parsing it, because a generic or a qualified type is outside the annotation grammar.

Read the values back with str, strs, bool, has and get. One annotation bound against four wrong specs, each refusal carrying the line and column of the annotation:

refusals for #app.get("/users", summary="List users", tags="users", tags="public")

1:1: there is no tags              // the spec never mentions tags
1:1: path takes true or false      // the spec asks for path as a Bool
1:1: tags is given more than once  // the spec lets tags be given once
1:1: method is required            // the spec requires a method

Step 7

A derive generator

A generator implements Generator: a name for the header, and the code to write for every declaration of its inputs. One call sees all of them, so a derive can answer per declaration and a registrar can gather every annotated function into one table.

Here is one that gives an annotated struct a describe method:

src/main.mbt

struct Describe {}

impl @gen.Generator for Describe with fn name(_) {
  "describe"
}

impl @gen.Generator for Describe with fn generate(_, decls) {
  let out = []
  for decl in decls {
    guard decl is Struct(name~, fields~, attrs~, ..) &&
      @attr.find(attrs, "demo.describe") is Some(_) else {
      continue
    }
    let text = name + "(" + fields.map(f => f.name).join(", ") + ")"
    let source =
      #|fn $T::describe(self : $T) -> String {
      #|  ignore(self)
      #|  $text
      #|}
    out.push(
      @quote.items(source)
      .set("T", @quote.ident(name))
      .set("text", @quote.string(text))
      .render(),
    )
  }
  @quote.join(out)
}

async fn main {
  @cli.run(Describe::{}, @env.args()[1:])
}

Its input is an ordinary file of declarations. The annotation is the only mark on it:

models.mbt

#demo.describe
pub struct Point {
  x : Int
  y : Int
}

Run it the way a dev_build rule would:

$ moon run . --target native -- models.mbt -o describe.g.mbt

and this is the whole file it writes:

describe.g.mbt

// Generated by describe from models.mbt. Edit those files, not this one.

///|
fn Point::describe(self : Point) -> String {
  ignore(self)
  "Point(x, y)"
}

An annotation that takes arguments

The same shape, reading a repeated skip off the annotation and using a field's type as a hole of its own:

impl @gen.Generator for Getters with fn generate(_, decls) {
  let out = []
  for decl in decls {
    guard decl is Struct(name~, fields~, attrs~, ..) else { continue }
    guard @attr.find(attrs, "demo.getters") is Some(annotation) else { continue }
    let skip = @attr.Spec::new("demo.getters")
      .many("skip", Ident)
      .bind(annotation)
      .strs("skip")
    for f in fields {
      guard !skip.contains(f.name) else { continue }
      let source =
        #|fn $T::$field(self : $T) -> $Ty {
        #|  self.$field
        #|}
      out.push(
        @quote.items(source)
        .set("T", @quote.ident(name))
        .set("field", @quote.ident(f.name))
        .set("Ty", @quote.ty(f.ty).render())
        .render(),
      )
    }
  }
  @quote.join(out)
}

models.mbt

#demo.getters(skip=token)
pub struct User {
  id : Int
  name : String
  token : String
}

getters.g.mbt

// Generated by getters from models.mbt. Edit those files, not this one.

///|
fn User::id(self : User) -> Int {
  self.id
}

///|
fn User::name(self : User) -> String {
  self.name
}

token has no getter, because the annotation asked for it to be skipped.

A template on its own

Nothing above needs a generator binary. A package that builds code can use a template directly, for instance in a test:

let source =
  #|fn $T::$field(self : $T) -> $Ty {
  #|  self.$field
  #|}
let code = @quote.items(source)
  .set("T", @quote.ident("User"))
  .set("field", @quote.ident("id"))
  .set("Ty", @quote.ty("Int").render())
println(code)

output

///|
fn User::id(self : User) -> Int {
  self.id
}

Step 8

Wire it into the build

Publish the generator as a module whose root package is that executable. Its users add one rule to moon.mod:

rule(name: "describe", command: "moonx <module>@<version> -C \"$mod_dir\" \"$input\" -o \"$output\"")

and one line to the moon.pkg of the package whose declarations it reads:

dev_build(rule: "describe", input: "models.mbt", output: "describe.g.mbt")
formatter(ignore: [ "describe.g.mbt" ])

From then on moon check, moon build and moon run regenerate the file whenever models.mbt changes.

-C matters. A rule does not run in the module directory, so pass -C "$mod_dir" and the paths moon gives you stay relative to the module.

Step 9

--check and stale files

A project that prefers to commit generated files runs the generator with --check. It writes nothing; it compares, and refuses when the output is stale.

$ moon run . --target native -- models.mbt -o describe.g.mbt --check
$ echo $?
0

# after somebody edits describe.g.mbt by hand
$ moon run . --target native -- models.mbt -o describe.g.mbt --check
describe.g.mbt is out of date with its inputs; run the generator again
$ echo $?
1

That exit code is the whole point: it turns "the generated file drifted" into a step that fails. A checkout that turned line feeds into CRLF has not changed the code, so it does not fail.

The command line is [-C <dir>] <inputs...> -o <output> [--check]. Anything else is refused, with the usage after the message:

-o is required; usage: [-C <dir>] <inputs...> -o <output> [--check]
no input files; usage: [-C <dir>] <inputs...> -o <output> [--check]

Step 10

Several files, one table

One call to generate sees every declaration of every input, so a registrar is the same shape as a derive — it just collects instead of answering per declaration. Given two route functions:

routes.mbt

#app.route("/users", tag="users", tag="public")
pub fn list_users() -> Array[String] {
  []
}

#app.route("/users", tag="users")
pub fn create_user(name : String) -> Unit {
  ignore(name)
}

impl @gen.Generator for Route with fn generate(_, decls) {
  let out = []
  for decl in decls {
    guard decl is Func(name~, attrs~, ..) else { continue }
    guard @attr.find(attrs, "app.route") is Some(annotation) else { continue }
    let args = @attr.Spec::new("app.route")
      .positional("path", Str)
      .many("tag", Str)
      .bind(annotation)
    let path = args.str("path").unwrap_or("/")
    let tags = args.strs("tag")
    let source =
      #|fn $reg(router : Router) -> Unit {
      #|  router.add($path, $handler, [$tags*])
      #|}
    out.push(
      @quote.items(source)
      .set("reg", @quote.ident("register_" + name))
      .set("path", @quote.string(path))
      .set("handler", @quote.ident(name))
      .splice("tags", tags.map(t => @quote.string(t)))
      .render(),
    )
  }
  @quote.join(out)
}

routes.g.mbt

// Generated by route from routes.mbt. Edit those files, not this one.

///|
fn register_list_users(router : Router) -> Unit {
  router.add("/users", list_users, ["users", "public"])
}

///|
fn register_create_user(router : Router) -> Unit {
  router.add("/users", create_user, ["users"])
}

Pass more than one input and the header names them all, and every declaration arrives with the file it came from:

header, two inputs

// Generated by route from a.mbt, b.mbt. Edit those files, not this one.

A run in which nothing matched is not an error: the file is written with its header and nothing under it, so a stale file still shrinks to the truth.

Reference

What the annotation grammar allows

The parser reads strings, true/false, names and nested calls as annotation arguments. There are no arrays and no numbers, a generic or qualified type has to be written as a string, and a keyword (method, type) cannot be a label. A list is written as a repeated key.

When a payload is outside that grammar, the annotation has no name as far as moonast is concerned, so find does not see it:

// #app.get("/users", 1) — a number is not an argument
inspect(@attr.find(attrs, "app.get") is Some(_), content="false")

Reference

Refusals

Every fallible call gives a Refused with a message and, where there is one, a position.

From	Means
`quote`	`the template is not MoonBit at 1:7: …`
`quote`	`hole $name was given nothing`
`quote`	`the template has no hole $x`
`quote`	`a-b is not an identifier`, `expected declarations`
`attr`	`1:1: cannot read the arguments of #app.get("/users", 1)`
`attr`	missing, unknown, mistyped, repeated — each with a line and column
`gen`	`models.mbt:3:5: …` when an input is not MoonBit
`cli`	a usage message, or `… is out of date with its inputs`

A generator's generate may raise, and whatever it raises travels out through render and run — so a refusal from a template or an annotation stops the run with its position intact.

Try it in the playground Browse the API