MoonBit · compression
Thirteen formats, one per package, each with the interface its own
specification argues for. DEFLATE, gzip, zlib, zip, tar, lz4, snappy,
LZMA, xz, bzip2, 7z, zstd, brotli — readers that check what they read,
writers that stream, and no dependency beyond moonbase.
One body — 184,000 bytes of English text with a periodic tail — compressed by
each format, printed by src/examples/stream and
src/examples/formats. The bars are on a
square-root scale, or everything below gzip would be invisible. Eight
compressing formats over one body, and a spread of four hundred to one: the
three at the bottom see repetition that a 32 KiB window cannot, and bzip2
pays for that with a sort, xz with a range coder, and brotli with a hash
chain long enough to find the match at 11,546 bytes.
Every package is named after the format it implements. Add the module, import
the package you need, and the function names are the ones the specification
uses — gzip, gunzip, tar,
untar.
import { "moonbitstack/moonzip" }
let packed = @gzip.gzip(b"the body"[:]) let back = @gzip.gunzip(packed[:])
// Nothing is held whole: the only state between calls is the 32 KiB window // the compressor needs and four bytes of running checksum. let writer = @gzip.Writer::new(level=9) for part in parts { for b in writer.push(part) { out.push(b) } } for b in writer.finish() { out.push(b) }
let archive = @zip.zip([ @zip.Entry::new("readme.txt", body), @zip.Entry::new("data/table.csv", table), ][:]) // or the same thing, one member at a time let w = @zip.Writer::new(password="a password") w.add(entry) // returns that member's bytes now w.finish() // returns the directory at the end
A format is either implemented or it is not; there is no "mostly". The table says which of the two, and the notes say what the hard part was. Nothing in the library is a stub that returns its input.
| Package | Specification | Read | Write | Notes |
|---|---|---|---|---|
| deflate | RFC 1951 | yes | yes | All three block types, both ways. The writer measures fixed, dynamic and stored per block and keeps the smallest. |
| gzip | RFC 1952 | yes | yes | Header fields, the header's own CRC, the trailing CRC-32 and length. Streaming via Writer/Reader. |
| zlib | RFC 1950 | yes | yes | Adler-32, the window the header names, the check bits. |
| zip | APPNOTE 6.3 | yes | yes | Zip64, data descriptors, a reader that walks local headers from the front, and volume splitting into .z01 pieces. ZipCrypto for old archives; WinZip AES (AE-1 and AE-2, 128/192/256) for real encryption. |
| tar | POSIX pax | yes | yes | ustar, pax extended headers, GNU long names on read. Symbolic links, directories, base-256 numbers. targz is the archive through gzip. |
| lz4 | block + frame | yes | yes | Both formats, with xxHash-32 for the header and content checksums and skippable frames stepped over. |
| snappy | block + framing v2 | yes | yes | Both formats, masked CRC-32C per chunk, skippable and unskippable chunk ranges both honoured. |
| lzma | LZMA SDK | yes | yes | The range coder and the probability model, and the .lzma container both ways — including the end-marker form some writers emit and ours never does. |
| xz | xz file format | yes | yes | LZMA2 chunks, CRC-32/CRC-64/none checks, the index and the footer. Verified both ways against an independent reference decoder. |
| bzip2 | bzip2 | yes | yes | Burrows-Wheeler, move-to-front and six Huffman tables, both directions — verified against the reference bzip2 implementation. |
| brotli | RFC 7932 | yes | yes | Block types, four context modes, and the 122 KB static dictionary with its 121 transforms — verified against an independent reference decoder at qualities 0 to 11. The writer is LZ77 with a quality from 0 to 11 that deepens the search and, at 10 and 11, chooses the parse outright: a walk over the block that keeps the cheapest price of reaching every position, priced from the codes the round before produced. |
| zstd | RFC 8878 | yes | yes | The full reader — FSE, Huffman literals in one or four streams, every table mode, repeat offsets — verified against an independent reference decoder. The writer uses the predefined tables and raw literals. |
| 7z | 7z format | yes | yes | Folders of coders over COPY, LZMA, LZMA2, BZIP2, DEFLATE and delta — read from real 7-Zip archives, including ones whose directory is itself compressed. AES, the branch filters and BCJ2 are refused by name. |
A format that is not implemented is refused by name, with the method number,
rather than guessed at. zip takes a Codec so a
method a caller has implemented elsewhere can be plugged in without an edit
to the package.
A response body arrives in chunks. A file is read four kilobytes at a time. A streaming interface means the caller never has to gather the whole thing, and the only state kept between calls is the window a back-reference can reach.
// compressor w.push(chunk) → the bytes that are ready (maybe none) w.finish() → the rest, plus the trailer // decompressor r.push(chunk) → the bytes it decoded, possibly across a symbol r.finish() → the rest, and the trailer is checked here
Every streaming test feeds the same body in one, two, three, five, seven, thirteen, sixty-four, five-hundred-and-eleven, four-thousand-and-ninety-six and sixty-five-thousand-five-hundred-and-thirty-six byte pieces, and requires the same bytes out each time. The boundaries that break a naive decoder are never the round ones.
Browsers carry the same algorithm this library implements. Type something and compress it with the platform's own DEFLATE — the numbers are the ones the format actually produces, not a simulation.
Runs in your browser through CompressionStream, which is the
same algorithm the platform ships. No input leaves this page.
The full interface is generated by moon info and lives beside
each package. This is the part a caller reads before writing anything — and
as one page in the API reference.
| Package | Function | Signature | Notes |
|---|---|---|---|
| root | crc32 · adler32 · crc32c | (BytesView) → UInt32 | And Crc, Adler, Castagnoli for a body taken a piece at a time. |
| root | mask_crc32c | (UInt32) → UInt32 | The mask Snappy puts on a CRC before a frame carries it. |
| deflate | deflate | (BytesView, level~=6, window~=32768, chain?, blocks~=Auto) → Bytes | Blocks is Auto, Fixed, Dynamic or Stored. |
| deflate | inflate | (BytesView, limit?) → Bytes raise | Refused carries the byte offset it gave up at. |
| deflate | Writer · Reader | push(BytesView) → Bytes · finish() → Bytes | The streaming pair the containers are built on. |
| gzip | gzip · gunzip | (BytesView, level~=6, mtime~=0, name?, comment?, os~=255) → Bytes | mtime is zero so the same input gives the same bytes. |
| gzip | Writer · Reader | new(level~, mtime~, name?, comment?, os~) · new(checksum~, length~, header_crc~, limit?) | Checks default on; a caller recovering a broken transfer turns them off. |
| zlib | zlib · unzlib | (BytesView, level~=6, window~=32768) → Bytes | The window goes in the header, so a reader learns it before it starts. |
| zip | zip | (ArrayView[Entry], level~, codecs~, zip64~, descriptor~, password?) → Bytes | Raises nothing: a writer has no input to distrust. |
| zip | unzip · names · entry | (BytesView, limit?, codecs~, password?) → Array[Entry] | names reads the directory without expanding anything. |
| zip | Writer · Reader | add(Entry) → Bytes · finish() → Bytes | And Reader::next walks local headers, descriptors and all. |
| tar | tar · untar · names | (ArrayView[Entry]) → Bytes | Entry carries mode, mtime, uid, gid, uname, gname and the link target. |
| tar | targz · untargz | (ArrayView[Entry], level~=6) → Bytes | A tarball is the archive through gzip, which is all .tar.gz ever was. |
| lz4 | lz4 · unlz4 | (BytesView, level~=0) → Bytes | The block format. level is the search depth; 0 is LZ4's own fast mode. |
| lz4 | frame · unframe | (BytesView) → Bytes | Blocks with a header, a content xxHash and skippable-frame support. |
| lz4 | Writer · Reader | new(level~=0) · new(checksum~=true, limit?) | A frame a block at a time; the content xxHash covers what the reader never holds. |
| snappy | snappy · unsnappy | (BytesView) → Bytes | The block format, with the length in front as a varint. |
| snappy | Writer · Reader | new() · new(checksum~=true, limit?) | A chunk is its own length, so a reader waits for exactly one. |
| lzma | Writer · Reader | new(level~=6) · new(limit?) | No blocks to wait for: the dictionary and the range coder carry across calls. |
| xz | xz · unxz | (BytesView, level?=6, check~=Crc64) → Bytes | Check is None, Crc32 or Crc64; the index and footer are checked. |
| xz | Writer · Reader | new(level~=6, check~=Crc64) · new(limit?) | A chunk at a time; the index is written at the end because its sizes are known then. |
| bzip2 | bzip2 · unbzip2 | (BytesView, level?=9) → Bytes | level is the block size in hundreds of kilobytes, 1 to 9. |
| bzip2 | Writer · Reader | push(BytesView) → Bytes · finish() → Bytes | Blocks are not padded, so the reader carries a bit position. |
| sevenz | sevenz · un7z | (ArrayView[Entry], level?=6, how?=Lzma2, solid?=true) → Bytes | names reads the file table without expanding anything. |
| zstd | zstd · unzstd | (BytesView, level?=3) → Bytes | Frames with a content size and a checksum; the writer uses the predefined FSE tables. |
| zstd | Writer · Reader | new(level~=3) · new(checksum~=true, limit?) | The reader keeps the window and the four tables the next block needs. |
| brotli | brotli · unbrotli | (BytesView, window~=22) → Bytes | window is the log of the window, 10 to 24. The reader knows all four context modes, every block type and the static dictionary. |
| brotli | Writer · Reader | new(window~=22, quality~=9) · new(limit?) | A meta-block at a time; a copy may still reach into the one before it. |
pub(all) suberror Refused { Truncated(at~ : Int32) // ends in the middle of something Header(at~ : Int32, what~ : String) // a field the format does not allow Code(at~ : Int32) // a code no symbol is assigned to Distance(at~ : Int32) // a back-reference before the start Checksum(want~ : UInt32, got~ : UInt32) Length(want~ : UInt32, got~ : Int32) TooBig(limit~ : Int32) // the caller's own bound }
A compressor that only ever reads its own output can be consistently wrong for years. Two scripts make that impossible to get away with, and CI runs both on every push.
scripts/conformance.py takes every stream
src/examples/vectors prints and hands it to an independent
reference decoder. Every combination of level, block type, chunk size,
Zip64, descriptor and password has to come back as the body it started
from.
scripts/interop.py writes archives with the platform's
standard archive writers — the plain form, one with data descriptors
because the sink it wrote to could not seek, a forced Zip64 header, and
tar in ustar, pax and GNU — and pins them as tests. They are the shapes
our own writer would never produce.
git diff --exit-code — formatting is not a matter of taste here.git diff --exit-code — the interface is checked in, so a change to it shows up in the diff.
Every one is a runnable executable under src/examples/, each
showing a different slice of the surface — from the smallest program that
works to a release packer that streams, wraps and reads back. Run any of them
with moon run src/examples/<name>.
| Example | What it shows | Run |
|---|---|---|
| minimal | The smallest program: compress one buffer with gzip and read it back. | moon run src/examples/minimal |
| tour | deflate, gzip, zlib and brotli round-tripped, the two checksums, and a tampered stream refused. | moon run src/examples/tour |
| formats | Every codec, one-shot, with the size it reached and a round-trip check. | moon run src/examples/formats |
| stream | The streaming pair, both ways, on a body taken a piece at a time. | moon run src/examples/stream |
| checksums | CRC-32, Adler-32 and CRC-32C, one-shot and streamed a piece at a time. | moon run src/examples/checksums |
| archive | zip (Zip64, descriptors, a password) and tar, tar.gz and 7z. | moon run src/examples/archive |
| bundle | A small release packer: stream-compress, wrap in four containers, read back. | moon run src/examples/bundle |
| vectors | The byte streams handed to an independent reference decoder for conformance. | moon run src/examples/vectors |