MoonBit · compression

Write bytes.
Read them back.

Thirteen formats, one per package, each with the interface its own specification argues for. DEFLATE, gzip, zlib, zip, tar, lz4, snappy, LZMA, xz, bzip2, 7z, zstd, brotli — readers that check what they read, writers that stream, and no dependency beyond moonbase.

raw184,000 B
snappy9,884 B
zlib2,667 B
deflate2,661 B
lz41,997 B
gzip1,983 B
bzip2853 B
brotli682 B
xz468 B

One body — 184,000 bytes of English text with a periodic tail — compressed by each format, printed by src/examples/stream and src/examples/formats. The bars are on a square-root scale, or everything below gzip would be invisible. Eight compressing formats over one body, and a spread of four hundred to one: the three at the bottom see repetition that a 32 KiB window cannot, and bzip2 pays for that with a sort, xz with a range coder, and brotli with a hash chain long enough to find the match at 11,546 bytes.

01 — InstallThree lines, then a stream.

Every package is named after the format it implements. Add the module, import the package you need, and the function names are the ones the specification uses — gzip, gunzip, tar, untar.

moon.mod
import { "moonbitstack/moonzip" }
a gzip round trip
let packed = @gzip.gzip(b"the body"[:])
let back   = @gzip.gunzip(packed[:])
a body that arrives in pieces
// Nothing is held whole: the only state between calls is the 32 KiB window
// the compressor needs and four bytes of running checksum.
let writer = @gzip.Writer::new(level=9)
for part in parts {
  for b in writer.push(part) { out.push(b) }
}
for b in writer.finish() { out.push(b) }
an archive
let archive = @zip.zip([
  @zip.Entry::new("readme.txt", body),
  @zip.Entry::new("data/table.csv", table),
][:])

// or the same thing, one member at a time
let w = @zip.Writer::new(password="a password")
w.add(entry)   // returns that member's bytes now
w.finish()     // returns the directory at the end

02 — The formatsWhat is here, and what is not.

A format is either implemented or it is not; there is no "mostly". The table says which of the two, and the notes say what the hard part was. Nothing in the library is a stub that returns its input.

PackageSpecificationReadWriteNotes
deflateRFC 1951 yesyes All three block types, both ways. The writer measures fixed, dynamic and stored per block and keeps the smallest.
gzipRFC 1952 yesyes Header fields, the header's own CRC, the trailing CRC-32 and length. Streaming via Writer/Reader.
zlibRFC 1950 yesyes Adler-32, the window the header names, the check bits.
zipAPPNOTE 6.3 yesyes Zip64, data descriptors, a reader that walks local headers from the front, and volume splitting into .z01 pieces. ZipCrypto for old archives; WinZip AES (AE-1 and AE-2, 128/192/256) for real encryption.
tarPOSIX pax yesyes ustar, pax extended headers, GNU long names on read. Symbolic links, directories, base-256 numbers. targz is the archive through gzip.
lz4block + frame yesyes Both formats, with xxHash-32 for the header and content checksums and skippable frames stepped over.
snappyblock + framing v2 yesyes Both formats, masked CRC-32C per chunk, skippable and unskippable chunk ranges both honoured.
lzmaLZMA SDK yesyes The range coder and the probability model, and the .lzma container both ways — including the end-marker form some writers emit and ours never does.
xzxz file format yesyes LZMA2 chunks, CRC-32/CRC-64/none checks, the index and the footer. Verified both ways against an independent reference decoder.
bzip2bzip2 yesyes Burrows-Wheeler, move-to-front and six Huffman tables, both directions — verified against the reference bzip2 implementation.
brotliRFC 7932 yesyes Block types, four context modes, and the 122 KB static dictionary with its 121 transforms — verified against an independent reference decoder at qualities 0 to 11. The writer is LZ77 with a quality from 0 to 11 that deepens the search and, at 10 and 11, chooses the parse outright: a walk over the block that keeps the cheapest price of reaching every position, priced from the codes the round before produced.
zstdRFC 8878 yesyes The full reader — FSE, Huffman literals in one or four streams, every table mode, repeat offsets — verified against an independent reference decoder. The writer uses the predefined tables and raw literals.
7z7z format yesyes Folders of coders over COPY, LZMA, LZMA2, BZIP2, DEFLATE and delta — read from real 7-Zip archives, including ones whose directory is itself compressed. AES, the branch filters and BCJ2 are refused by name.

A format that is not implemented is refused by name, with the method number, rather than guessed at. zip takes a Codec so a method a caller has implemented elsewhere can be plugged in without an edit to the package.

03 — StreamingThe body never has to be in one piece.

A response body arrives in chunks. A file is read four kilobytes at a time. A streaming interface means the caller never has to gather the whole thing, and the only state kept between calls is the window a back-reference can reach.

the shape
// compressor
w.push(chunk)   → the bytes that are ready (maybe none)
w.finish()      → the rest, plus the trailer

// decompressor
r.push(chunk)   → the bytes it decoded, possibly across a symbol
r.finish()      → the rest, and the trailer is checked here
what a chunk boundary may fall inside
  • A header field. Status lines, file names, pax records. The reader gathers until the header is complete, with a bound so a stream that never ends is refused rather than buffered forever.
  • A Huffman code. The decoder never consumes half a symbol and waits: eight bytes of buffered input cover the widest step, and with fewer than that it stops.
  • A stored block. The payload is copied straight out of the buffer, and a chunk boundary in the middle of one is ordinary.
  • A data descriptor. When an entry's sizes follow its data, the DEFLATE stream itself says where it ended, and the descriptor is read from there.
what the tests do

Every streaming test feeds the same body in one, two, three, five, seven, thirteen, sixty-four, five-hundred-and-eleven, four-thousand-and-ninety-six and sixty-five-thousand-five-hundred-and-thirty-six byte pieces, and requires the same bytes out each time. The boundaries that break a naive decoder are never the round ones.

04 — In the browserDEFLATE, where you can watch it.

Browsers carry the same algorithm this library implements. Type something and compress it with the platform's own DEFLATE — the numbers are the ones the format actually produces, not a simulation.

compressed, then read back
press compress

Runs in your browser through CompressionStream, which is the same algorithm the platform ships. No input leaves this page.

05 — InterfacesOne page, every entry point.

The full interface is generated by moon info and lives beside each package. This is the part a caller reads before writing anything — and as one page in the API reference.

PackageFunctionSignatureNotes
rootcrc32 · adler32 · crc32c(BytesView) → UInt32And Crc, Adler, Castagnoli for a body taken a piece at a time.
rootmask_crc32c(UInt32) → UInt32The mask Snappy puts on a CRC before a frame carries it.
deflatedeflate(BytesView, level~=6, window~=32768, chain?, blocks~=Auto) → BytesBlocks is Auto, Fixed, Dynamic or Stored.
deflateinflate(BytesView, limit?) → Bytes raiseRefused carries the byte offset it gave up at.
deflateWriter · Readerpush(BytesView) → Bytes · finish() → BytesThe streaming pair the containers are built on.
gzipgzip · gunzip(BytesView, level~=6, mtime~=0, name?, comment?, os~=255) → Bytesmtime is zero so the same input gives the same bytes.
gzipWriter · Readernew(level~, mtime~, name?, comment?, os~) · new(checksum~, length~, header_crc~, limit?)Checks default on; a caller recovering a broken transfer turns them off.
zlibzlib · unzlib(BytesView, level~=6, window~=32768) → BytesThe window goes in the header, so a reader learns it before it starts.
zipzip(ArrayView[Entry], level~, codecs~, zip64~, descriptor~, password?) → BytesRaises nothing: a writer has no input to distrust.
zipunzip · names · entry(BytesView, limit?, codecs~, password?) → Array[Entry]names reads the directory without expanding anything.
zipWriter · Readeradd(Entry) → Bytes · finish() → BytesAnd Reader::next walks local headers, descriptors and all.
tartar · untar · names(ArrayView[Entry]) → BytesEntry carries mode, mtime, uid, gid, uname, gname and the link target.
tartargz · untargz(ArrayView[Entry], level~=6) → BytesA tarball is the archive through gzip, which is all .tar.gz ever was.
lz4lz4 · unlz4(BytesView, level~=0) → BytesThe block format. level is the search depth; 0 is LZ4's own fast mode.
lz4frame · unframe(BytesView) → BytesBlocks with a header, a content xxHash and skippable-frame support.
lz4Writer · Readernew(level~=0) · new(checksum~=true, limit?)A frame a block at a time; the content xxHash covers what the reader never holds.
snappysnappy · unsnappy(BytesView) → BytesThe block format, with the length in front as a varint.
snappyWriter · Readernew() · new(checksum~=true, limit?)A chunk is its own length, so a reader waits for exactly one.
lzmaWriter · Readernew(level~=6) · new(limit?)No blocks to wait for: the dictionary and the range coder carry across calls.
xzxz · unxz(BytesView, level?=6, check~=Crc64) → BytesCheck is None, Crc32 or Crc64; the index and footer are checked.
xzWriter · Readernew(level~=6, check~=Crc64) · new(limit?)A chunk at a time; the index is written at the end because its sizes are known then.
bzip2bzip2 · unbzip2(BytesView, level?=9) → Byteslevel is the block size in hundreds of kilobytes, 1 to 9.
bzip2Writer · Readerpush(BytesView) → Bytes · finish() → BytesBlocks are not padded, so the reader carries a bit position.
sevenzsevenz · un7z(ArrayView[Entry], level?=6, how?=Lzma2, solid?=true) → Bytesnames reads the file table without expanding anything.
zstdzstd · unzstd(BytesView, level?=3) → BytesFrames with a content size and a checksum; the writer uses the predefined FSE tables.
zstdWriter · Readernew(level~=3) · new(checksum~=true, limit?)The reader keeps the window and the four tables the next block needs.
brotlibrotli · unbrotli(BytesView, window~=22) → Byteswindow is the log of the window, 10 to 24. The reader knows all four context modes, every block type and the static dictionary.
brotliWriter · Readernew(window~=22, quality~=9) · new(limit?)A meta-block at a time; a copy may still reach into the one before it.
errors
pub(all) suberror Refused {
  Truncated(at~ : Int32)                   // ends in the middle of something
  Header(at~ : Int32, what~ : String)      // a field the format does not allow
  Code(at~ : Int32)                        // a code no symbol is assigned to
  Distance(at~ : Int32)                    // a back-reference before the start
  Checksum(want~ : UInt32, got~ : UInt32)
  Length(want~ : UInt32, got~ : Int32)
  TooBig(limit~ : Int32)                   // the caller's own bound
}

06 — ProofAgreeing with itself is not enough.

A compressor that only ever reads its own output can be consistently wrong for years. Two scripts make that impossible to get away with, and CI runs both on every push.

others read what we write

scripts/conformance.py takes every stream src/examples/vectors prints and hands it to an independent reference decoder. Every combination of level, block type, chunk size, Zip64, descriptor and password has to come back as the body it started from.

we read what others write

scripts/interop.py writes archives with the platform's standard archive writers — the plain form, one with data descriptors because the sink it wrote to could not seek, a forced Zip64 header, and tar in ustar, pax and GNU — and pins them as tests. They are the shapes our own writer would never produce.

the gate
  • moon check --target all --deny-warn — four backends, and a warning is an error.
  • moon test --target all — the same tests on wasm, wasm-gc, js and native.
  • moon fmt, then git diff --exit-code — formatting is not a matter of taste here.
  • moon info, then git diff --exit-code — the interface is checked in, so a change to it shows up in the diff.

07 — ExamplesEight programs that use the library.

Every one is a runnable executable under src/examples/, each showing a different slice of the surface — from the smallest program that works to a release packer that streams, wraps and reads back. Run any of them with moon run src/examples/<name>.

ExampleWhat it showsRun
minimalThe smallest program: compress one buffer with gzip and read it back.moon run src/examples/minimal
tourdeflate, gzip, zlib and brotli round-tripped, the two checksums, and a tampered stream refused.moon run src/examples/tour
formatsEvery codec, one-shot, with the size it reached and a round-trip check.moon run src/examples/formats
streamThe streaming pair, both ways, on a body taken a piece at a time.moon run src/examples/stream
checksumsCRC-32, Adler-32 and CRC-32C, one-shot and streamed a piece at a time.moon run src/examples/checksums
archivezip (Zip64, descriptors, a password) and tar, tar.gz and 7z.moon run src/examples/archive
bundleA small release packer: stream-compress, wrap in four containers, read back.moon run src/examples/bundle
vectorsThe byte streams handed to an independent reference decoder for conformance.moon run src/examples/vectors