MoonBit · the guide

From one buffer
to an archive.

moonzip is one package per format. This walks the path a caller actually takes: compress a buffer, put buffers in an archive, compress as they arrive, and choose the format that fits. Every snippet below runs as written; its output is what the library printed.

01 — Deflate & gzipThe smallest unit is a buffer.

deflate is the raw codec (RFC 1951). gzip wraps it with a ten-byte header, a CRC-32 and a length (RFC 1952). zlib wraps it with a two-byte header and an Adler-32 (RFC 1950). All three round-trip to the byte.

compress & read back
moonbit
fn main raise {
  let out : Array[Byte] = []
  for _ in 0..<200 {
    for b in "the quick brown fox jumps over the lazy dog. " {
      out.push((b.to_int() % 256).to_byte())
    }
  }
  let body = Bytes::from_array(out)
  println("original  -> \{body.length()} bytes")

  let raw = @deflate.deflate(body[:])
  println("deflate   -> \{raw.length()} bytes, back to \{@deflate.inflate(raw[:]).length()}")

  let gz = @gzip.gzip(body[:])
  println("gzip      -> \{gz.length()} bytes, back to \{@gzip.gunzip(gz[:]).length()}")

  let zl = @zlib.zlib(body[:])
  println("zlib      -> \{zl.length()} bytes, back to \{@zlib.unzlib(zl[:]).length()}")
}
output
original  -> 9000 bytes
deflate   -> 87 bytes, back to 9000
gzip      -> 105 bytes, back to 9000
zlib      -> 93 bytes, back to 9000
a tampered stream

A container that carries a checksum checks it. Flip one byte of the trailing CRC and gunzip refuses, rather than hand back bytes that look right and are not.

moonbit
fn main raise {
  let out : Array[Byte] = []
  for _ in 0..<200 {
    for b in "the quick brown fox jumps over the lazy dog. " {
      out.push((b.to_int() % 256).to_byte())
    }
  }
  let body = Bytes::from_array(out)

  let gz = @gzip.gzip(body[:])
  let bent = gz.to_array()
  bent[gz.length() - 8] = (bent[gz.length() - 8].to_int() ^ 0xFF).to_byte()
  try @gzip.gunzip(Bytes::from_array(bent)[:]) |> ignore catch {
    Checksum(..) => println("tampered  -> refused: the checksum does not match")
    _ => println("tampered  -> refused")
  } noraise {
    _ => println("tampered  -> accepted, which would be the bug")
  }
}
output
tampered  -> refused: the checksum does not match

The same input gives the same bytes every time: gzip writes a zero modification time by default, so a build stays reproducible.

02 — Archivesgzip holds one thing; zip and tar hold many.

zip is random-access: a central directory at the end names every member, so listing reads that and stops. tar is one header per file and no compressed members; targz is a tarball through gzip, which is all a .tar.gz ever was.

build & read
moonbit
fn main raise {
  let entries = [
    @zip.Entry::new("readme.txt", b"a member of an archive\n", mtime=1700000000L),
    @zip.Entry::new("data/table.csv", b"name,size\nreadme,24\n"),
  ]
  let archive = @zip.zip(entries[:])
  println("zip       -> \{archive.length()} bytes, \{@zip.names(archive[:]).length()} members")
  println("zip names -> \{@zip.names(archive[:]).join(", ")}")
  let back = @zip.unzip(archive[:])
  println("unzip     -> \{back.length()} members, first is \{back[0].name}")

  let files = [
    @tar.Entry::new("readme.txt", b"a member of an archive\n", mtime=1700000000L),
    @tar.Entry::new("data/table.csv", b"name,size\nreadme,24\n"),
  ]
  let tarball = @tar.tar(files[:])
  let gzipped = @tar.targz(files[:])
  println("tar       -> \{tarball.length()} bytes")
  println("tar.gz    -> \{gzipped.length()} bytes")
}
output
zip       -> 265 bytes, 2 members
zip names -> readme.txt, data/table.csv
unzip     -> 2 members, first is readme.txt
tar       -> 3072 bytes
tar.gz    -> 170 bytes

Listing reads only the directory; unzip expands on demand, so a caller can pull one member without touching the rest.

A member can be encrypted. ZipCrypto is the cipher the format called encryption for fifteen years, and it is broken; WinZip AES is the real one — counter mode under a PBKDF2-HMAC-SHA1 key, authenticated with an HMAC-SHA1 tag, at 128, 192 or 256 bits, in the AE-1 spelling that keeps the CRC or AE-2 that drops it. A split archive is the same bytes in fixed-size pieces, archive.z01 through archive.zip, that rejoin into an ordinary archive.

encrypt & split
moonbit
fn main raise {
  let entries = [@zip.Entry::new("secret.txt", b"attack at dawn")]

  let sealed = @zip.zip(entries[:], password="hunter2", aes=256)
  let opened = @zip.unzip(sealed[:], password="hunter2")
  println("aes-256  -> \{opened[0].name}, \{opened[0].data.length()} bytes")

  let parts = @zip.zip_split(entries[:], 64)
  println("volumes  -> \{parts.length()} pieces: \{@zip.volume_names("archive", parts.length()).join(", ")}")
  let joined = @zip.unzip_split(parts)
  println("rejoined -> \{joined[0].name}, \{joined[0].data.length()} bytes")
}
output
aes-256  -> secret.txt, 14 bytes
volumes  -> 3 pieces: archive.z01, archive.z02, archive.zip
rejoined -> secret.txt, 14 bytes

A wrong password is caught by the two-byte verification value, and a tampered payload by the tag, both before any byte is returned.

03 — StreamingThe body never has to be in one piece.

A response body arrives in chunks; a file is read four kilobytes at a time. The Writer/Reader pair takes it a piece at a time and never holds the whole thing — only the 32 KiB window a back-reference can reach, plus four bytes of running checksum. push returns what is ready; finish flushes the rest and checks the trailer.

Every codec carries the pair. What each one holds between calls is its own unit — a block for deflate, lz4, zstd and a brotli meta-block, a chunk for snappy and xz, a block for bzip2 — and lzma, which has no blocks at all, holds the dictionary a copy can reach into and carries its range coder from token to token.

4 KiB out, 777 B in
moonbit
fn main raise {
  let out : Array[Byte] = []
  for i in 0..<4000 {
    for b in "the quick brown fox jumps over the lazy dog. " {
      out.push((b.to_int() % 256).to_byte())
    }
    out.push((i % 251).to_byte())
  }
  let source = Bytes::from_array(out)
  println("body      -> \{source.length()} bytes")

  let writer = @gzip.Writer::new(level=9)
  let packed : Array[Byte] = []
  let mut at = 0
  while at < source.length() {
    let stop = if source.length() - at < 4096 { source.length() } else { at + 4096 }
    for b in writer.push(source[at:stop]) { packed.push(b) }
    at = stop
  }
  for b in writer.finish() { packed.push(b) }
  let gzipped = Bytes::from_array(packed)
  println("gzip      -> \{gzipped.length()} bytes (\{100 - gzipped.length() * 100 / source.length()}% smaller)")

  let reader = @gzip.Reader::new()
  let got : Array[Byte] = []
  let mut at2 = 0
  while at2 < gzipped.length() {
    let stop = if gzipped.length() - at2 < 777 { gzipped.length() } else { at2 + 777 }
    for b in reader.push(gzipped[at2:stop]) { got.push(b) }
    at2 = stop
  }
  for b in reader.finish() { got.push(b) }
  println("gunzip    -> \{Bytes::from_array(got).length()} bytes, matches: \{Bytes::from_array(got) == source}")
}
output
body      -> 184000 bytes
gzip      -> 1983 bytes (99% smaller)
gunzip    -> 184000 bytes, matches: true

The home page runs the same algorithm in the browser; here it is in MoonBit, on a body fed back in pieces that fall inside headers and codes.

04 — Choosing a formatNo format wins.

The numbers are one 184,000-byte body of English text with a periodic tail, printed by src/examples/stream and src/examples/formats. They are the same body the home page's strata show — here as a table, with what each format is for.

FormatBytesOf originalReach for it when
gzip1,9831.1%A single file on a wire — HTTP, a log, a download.
zlib2,6671.4%The deflate HTTP content coding; a stream wrapped by someone else's framing.
deflate (dynamic)2,6611.4%Raw DEFLATE when no container is wanted at all.
lz4 block1,9971.1%Decode speed above everything — a cache, a transport.
lz4 frame4,5762.5%lz4 with a header, a content checksum and skippable frames.
snappy block9,8845.4%The framing Google uses — protobuf on the wire, Parquet on disk.
snappy frame12,4396.8%Snappy with per-chunk masked CRC-32C.
brotli6820.4%Static assets on the web; it is the codec that serves this page's text.
zstd3,0421.7%A general replacement for gzip and zlib, with a fast decoder.
bzip28530.5%When the setup cost is worth it on highly repetitive text.
xz4680.3%Distribution tarballs, where size matters most and decode time does not.
lzma alone4150.2%The codec xz and 7z are built on, on its own.
— nothing184,000100%The body, uncompressed, for reference.

brotli takes a quality from 0 to 11 — higher is slower and never larger — and the row above is quality 9, the default, which already models literals by context, splits the command and literal streams into two block types each when the cut pays, and codes words the static dictionary knows as references rather than letters. Quality 10 and 11 add the one thing 9 does not do: they choose the parse instead of taking it, walking the block for the cheapest path to every position and keeping whichever round encoded smallest.

Pick by what the caller does with the bytes. Servicing a request: lz4 or snappy. Storing a file people download: gzip or zstd. Shipping a tarball: gzip for speed, xz for size. Every function above is documented in the API reference.