The guide

From one line of output to a hub that keeps writing while one of its sinks is down, and then out to the world: a console, a file that rotates, a push to Loki, a letter, a deployment file and the one process-wide hub. Every snippet on this page was compiled; the examples in the repository are the same code with more words around it.

One line

A logger needs somewhere to write. The smallest useful sink is Lines, which hands each rendered line to a function — so a program that prints is a program that logs.

let hub = @moonlog.Hub::new(clock=fn() { 1_700_000_000_000_000_000L })
let sink = @moonlog.Lines::new(fn(line) { println(line) })
let _ = hub.add(@moonlog.Slot::new(sink, name="console"))
hub.logger("app").info("listening")
// 2023-11-14 22:13:20.000 INFO  app: listening

The default line is {time} {level:<5} {target}: {msg}{fields}: local time with milliseconds, the level padded to five columns, where the record came from, the message, then the fields. Nothing is coloured unless you say so.

Levels and filters

Six levels ship: trace, debug, info, warn, error, fatal. They compare by number, so a level you define yourself — 35, say, between info and warn — sorts where it should.

let notice : @moonlog.Level = {
  name: "notice", value: 35, colour: @moonlog.Colour::Blue,
}
let quiet = @moonlog.Filter::parse("warn, app=debug, app::net=off")
quiet.allows(@moonlog.debug, "app::sync")   // true
quiet.allows(@moonlog.error, "app::net")   // false
quiet.lowest()                                // Some(debug)

A target matches itself and everything under it, split at ::, ., : or /; the most specific directive wins. The hub's filter decides once for every slot; a slot may then narrow it further with .at(level) or .filter(f).

let hub = @moonlog.Hub::new(
  clock=fn() { 0L },
  filter=@moonlog.Filter::parse("info,mirror::sync=debug"),
)
hub.set_level(@moonlog.warn)              // replace it at run time
hub.enabled(@moonlog.debug)                  // ask before building a message

Fields keep their type

A field is a (String, Value) pair. Values are typed — null, bool, int, float, string, list, map, error — so the layout, not the call site, decides how they read. That is what makes redaction by name and JSON output possible from the same record.

let log = hub.logger("sync")
log.info("gave up", fields=[
  ("error", @moonlog.Value::of_error(message="connect", causes=["refused"])),
  ("tries", @moonlog.Value::of_int(3)),
  ("req", @moonlog.Value::of_map([("id", @moonlog.Value::of_int(7))])),
])
// sync: gave up error=connect: refused tries=3 req.id=7

A map is spread into req.id=7 on a text line and kept as an object in JSON. An error keeps its causes: connect: refused on a line, {"message":"connect","causes":["refused"]} in a document. A later field of the same name replaces the earlier one where it stands, so scope comes before logger, and logger before the call.

Steps in a chain

A Processor may change a record or drop it. The hub's chain runs first, then each slot's own. Eight are built in, and a step of your own is one method.

let slot = @moonlog.Slot::new(sink, name="audit")
  .process(@moonlog.Redact::new().key("authorization").path("req.headers.auth"))
  .process(@moonlog.Sanitise::new())
  .process(@moonlog.Sample::new(first=2, every=10, per=60_000_000_000L))

// Your own: return the record, a changed copy, or None to drop it.
struct Region { }

pub impl @moonlog.Processor for Region with process(_self, record, _env) {
  Some(record.set("region", @moonlog.Value::of_str("eu")))
}

The three that need a moment get one: process(record, env), where env.now is the hub's instant in nanoseconds. No step reads a clock, so a test can drive a minute of sampling in a loop.

Layouts

A layout turns a record into one line. A pattern is literal text with holes; every token but time may be padded (<5, >5, ^7) and truncated (.4).

let line = @moonlog.Pattern::parse("{time:%H:%M:%S%.3f} {level:<5} {target:~24}: {msg}{fields}")
let json = @moonlog.JsonLine::new(@moonlog.Schema::Ecs)
let custom = @moonlog.JsonLine::new(@moonlog.Schema::Serlog)
  .rename("msg", "message")
  .nest("data")

// A preset name stands wherever a pattern does.
@moonlog.Pattern::parse("logback")

Sinks

A sink is one method. That is the whole contract: entries arrive already rendered, and a failure is a string.

struct Counter { mut lines : Int }

pub impl @moonlog.Sink for Counter with write(self, entry) {
  let _ = entry
  self.lines = self.lines + 1
  Ok(())
}

let counted = Counter::{ lines: 0 }
let _ = hub.add(@moonlog.Slot::new(counted, name="count"))

Lines takes a function, Memory keeps the newest entries so a test can read what was written, Discard throws them away. Nothing in this package opens a file or a socket: those are sinks too, and a sink you write is a sink you can test.

Slots

A slot wraps one sink with everything the sink should not know: filter, steps, layout, colour, queue, retries, probe interval and what to do on failure. kind names what the sink is, so a status page can say more than custom.

let _ = hub.add(
  @moonlog.Slot::new(sink, name="archive")
  .kind("file")
  .at(@moonlog.info)
  .layout(@moonlog.JsonLine::new(@moonlog.Schema::Serlog))
  .queue(@moonlog.Queue::new(size=4096).batch(200).wait(1_000_000_000L))
  .retries(3)
  .probe(5_000_000_000L),
)

for slot in hub.status() {
  println(slot.name + " (" + slot.kind + "): " + slot.health.name().to_owned())
}

A queue holds records until it has batch of them, until wait has passed since the first, or until hub.flush(). With no runtime to hand behind it, the hub drains it — the same policy a background task would follow, with the clock still yours. When it is full, Full::Drop counts the loss and Full::Block writes first.

When a sink fails

A failed write moves the breaker to down. Records for that slot are counted and dropped until its next probe; the probe is one write, and it decides. Repeated failures double the wait, up to five minutes; a success puts the slot back and starts the wait over.

let oncall = @moonlog.Tiers::from([["loki"], ["mail"], ["stderr"]])
  .with_fallback(@moonlog.Fallback::Ignore)
  .with_remind(1_800_000_000_000L)

let _ = hub.add(
  @moonlog.Slot::new(sink, name="loki").on_fail(@moonlog.OnFail::Alarm(oncall)),
)

An incident is delivered through the tiers in order. The failing sink is never asked, a sink that is resting is left out, and a sink already tried is not tried twice. * stands for every sink on the log route. The walk stops at the first tier where one sink took the record, and the incident record carries sink, phase, failures and what was lost.

OnFail::Ignore counts and says nothing; OnFail::Panic stops the program.

Profiles

A profile is four defaults at once: the line, the console stream, the starting level and which variable a filter is read from. It exists so that a deployment moving from another logger does not have to relearn its own conventions.

let hub = @moonlog.Hub::new(profile=@moonlog.Profile::Python, clock=fn() { 0L })
hub.logger("app").warn("careful")
// WARNING:app:careful

stdio

The six subpackages are where a logger meets the world. The first is the console: three streams, and an auto that sends a record to standard error when its level reaches error and to standard output otherwise, so a pipeline that redirects the log still sees failures on the terminal.

let console = @stdio.Console::new(stream=@moonlog.Stream::Auto)
console.target(@moonlog.info)   // Stdout
console.target(@moonlog.error)  // Stderr
console.coloured()              // false: a pipe is not a terminal
let _ = hub.add(console.slot()) // a slot, already coloured to match

Whether a line carries escapes is settled once, when the console is built, from FORCE_COLOR, then NO_COLOR, then whether the stream is a terminal. Console::slot hands the answer to the slot, which is the only thing that decides how a line is written. On a backend with no file descriptor the same calls print through the host.

file

A file sink is written for the day it fills a disk. It opens on the first write, so a path it cannot open is a failing slot the health machine handles, not a panic at startup; it cuts itself into segments by size or by age; and it keeps the segments a policy allows.

let file = @file.File::new(
  path="/var/log/gateway/app.log",
  mkdir=true,
  rotation=@file.Rotation::new(size=64_000_000L, every=86_400_000_000_000L),
  limits=@file.Limits::new(keep=7, max_age=2_592_000_000_000_000L),
  compress=true,
)
let _ = hub.add(file.slot())
// app.log.20231114T221320.gz
file.segments()

A segment is named after the instant it was cut, in UTC, so a directory listing reads as a history: keep drops the oldest beyond a count, max_age the ones older than a span, max_total the oldest until the rest fit. The sequence continues past the highest number in the directory, so a pruned name is never handed out twice. compress packs a segment as it is cut.

reopen is for the case where something outside the program moved the file: the old descriptor would keep writing to a file nobody reads. flush and close are for a sink given a buffer, and shutdown in global calls them for you.

loki

A network sink cannot write inside a log call without making every caller wait for a socket. So write only queues, and push does the sending — from the async loop the application already runs.

let loki = @loki.Loki::new(
  url="http://loki:3100", labels=[("job", "gateway")],
)
let _ = hub.add(loki.slot())
// once per turn of a loop you already have
match loki.push() {
  report if report.pushed() => ()          // landed
  report => // wait `report.retry` said what, then come back
}

Each level becomes a stream of its own, so a query selects on level without parsing a line, and the record's fields travel as structured metadata. A status is read three ways: 2xx landed, 400 and the like will never land and those entries are dropped and counted, and 408, 429 and 5xx are worth another try — the batch goes back at the front of the queue, in order.

mail

The mail sink is the one a pager is on the other end of. It mails what a level floor lets through, gathers a burst into one letter, and caps how many records a letter carries.

let mail = @mail.Mail::new(
  host="smtp.example.com",
  from="gateway@example.com",
  to=["oncall@example.com"],
  level=@moonlog.error,
  window=60_000_000_000L, at_most=100,
)
let _ = hub.add(mail.slot())
if mail.due(now) {
  match mail.send() { ok => (), error => // stays queued () }
}

The subject is a template — {level}, {records}, {count}, {target} — and the body is a text part and a table in HTML, with the fields that came with each record. draft builds the letter without sending it, which is what a check before a deployment prints. STARTTLS and TLS need a tls client: which roots to trust is the application's decision, not this sink's.

config

A deployment file is read, checked, and turned into sinks. Reading builds nothing: Config says what was asked for, and global — or your own code — acts on it.

# filter: info, gateway=debug
# slots:
#   console: {kind: stdio, stream: auto}
#   app: {kind: file, path: /var/log/gateway/app.log, max_size: 64mb}

let config = @config.Config::load(path="/etc/gateway.yaml")
config.slots().length()                     // 2
config.dump(format=@config.Format::Toml)     // the same, in TOML

YAML, JSON and TOML read into the same tree, so the same document means the same configuration in all three. Every key is checked against what its kind of sink reads: a typo is refused with the path it sits at — slots.app.max_sise — before the process starts serving. 10mb, 1GiB, 200ms, 1m30s and 24h are understood where a file expects a size or a span.

Config::watch hands back a watcher whose poll answers the new document once the file on disk is newer than the one that was read: Ok(None) for nothing moved, and an error when the new file will not do, so a program keeps the configuration it has rather than the one that does not parse.

global

The root package has no process-wide hub, on purpose: a library takes a Logger and writes through it. An application, though, has exactly one place to configure, and that place is moonlog/global.

@global.start(config=@config.Config::load(path="/etc/gateway.yaml"))
@global.logger("gateway").info("serving")
for status in @global.status() {
  // name, kind, health, skipped, dropped, error
}
@global.flush()     // before the process goes away
@global.shutdown()

With no document, start sets up a console at the profile's own stream and level — and so does the first call to logger, which is why a program that says nothing still logs. SERLOG_COMPAT picks the profile, SERLOG_FILTER the filter and SERLOG_FORMAT the layout: logfmt, pretty, json, json:pino, pattern:…, or a pattern written out. drain is the loop for the network sinks: it pushes every Loki sink and sends every mail that is due.

Where to go next

The API reference lists every name with its parameters. Inside explains why the pieces are shaped this way — the dispatch order, the time model, the health machine and the alarm walk. The repository holds three runnable examples under src/examples.