MoonBit · the guide
moonzip is one package per format. This walks the path a caller actually takes: compress a buffer, put buffers in an archive, compress as they arrive, and choose the format that fits. Every snippet below runs as written; its output is what the library printed.
deflate is the raw codec (RFC 1951). gzip wraps it with
a ten-byte header, a CRC-32 and a length (RFC 1952). zlib wraps it
with a two-byte header and an Adler-32 (RFC 1950). All three round-trip to the
byte.
fn main raise {
let out : Array[Byte] = []
for _ in 0..<200 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
}
let body = Bytes::from_array(out)
println("original -> \{body.length()} bytes")
let raw = @deflate.deflate(body[:])
println("deflate -> \{raw.length()} bytes, back to \{@deflate.inflate(raw[:]).length()}")
let gz = @gzip.gzip(body[:])
println("gzip -> \{gz.length()} bytes, back to \{@gzip.gunzip(gz[:]).length()}")
let zl = @zlib.zlib(body[:])
println("zlib -> \{zl.length()} bytes, back to \{@zlib.unzlib(zl[:]).length()}")
}
original -> 9000 bytes deflate -> 87 bytes, back to 9000 gzip -> 105 bytes, back to 9000 zlib -> 93 bytes, back to 9000
A container that carries a checksum checks it. Flip one byte of the
trailing CRC and gunzip refuses, rather than hand back bytes
that look right and are not.
fn main raise {
let out : Array[Byte] = []
for _ in 0..<200 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
}
let body = Bytes::from_array(out)
let gz = @gzip.gzip(body[:])
let bent = gz.to_array()
bent[gz.length() - 8] = (bent[gz.length() - 8].to_int() ^ 0xFF).to_byte()
try @gzip.gunzip(Bytes::from_array(bent)[:]) |> ignore catch {
Checksum(..) => println("tampered -> refused: the checksum does not match")
_ => println("tampered -> refused")
} noraise {
_ => println("tampered -> accepted, which would be the bug")
}
}
tampered -> refused: the checksum does not match
The same input gives the same bytes every time: gzip writes a
zero modification time by default, so a build stays reproducible.
zip is random-access: a central directory at the end names every
member, so listing reads that and stops. tar is one header per file
and no compressed members; targz is a tarball through gzip, which is
all a .tar.gz ever was.
fn main raise {
let entries = [
@zip.Entry::new("readme.txt", b"a member of an archive\n", mtime=1700000000L),
@zip.Entry::new("data/table.csv", b"name,size\nreadme,24\n"),
]
let archive = @zip.zip(entries[:])
println("zip -> \{archive.length()} bytes, \{@zip.names(archive[:]).length()} members")
println("zip names -> \{@zip.names(archive[:]).join(", ")}")
let back = @zip.unzip(archive[:])
println("unzip -> \{back.length()} members, first is \{back[0].name}")
let files = [
@tar.Entry::new("readme.txt", b"a member of an archive\n", mtime=1700000000L),
@tar.Entry::new("data/table.csv", b"name,size\nreadme,24\n"),
]
let tarball = @tar.tar(files[:])
let gzipped = @tar.targz(files[:])
println("tar -> \{tarball.length()} bytes")
println("tar.gz -> \{gzipped.length()} bytes")
}
zip -> 265 bytes, 2 members zip names -> readme.txt, data/table.csv unzip -> 2 members, first is readme.txt tar -> 3072 bytes tar.gz -> 170 bytes
Listing reads only the directory; unzip expands on demand, so a
caller can pull one member without touching the rest.
A member can be encrypted. ZipCrypto is the cipher the format called encryption
for fifteen years, and it is broken; WinZip AES is the real one — counter mode
under a PBKDF2-HMAC-SHA1 key, authenticated with an HMAC-SHA1 tag, at 128, 192
or 256 bits, in the AE-1 spelling that keeps the CRC or AE-2 that drops it. A
split archive is the same bytes in fixed-size pieces, archive.z01
through archive.zip, that rejoin into an ordinary archive.
fn main raise {
let entries = [@zip.Entry::new("secret.txt", b"attack at dawn")]
let sealed = @zip.zip(entries[:], password="hunter2", aes=256)
let opened = @zip.unzip(sealed[:], password="hunter2")
println("aes-256 -> \{opened[0].name}, \{opened[0].data.length()} bytes")
let parts = @zip.zip_split(entries[:], 64)
println("volumes -> \{parts.length()} pieces: \{@zip.volume_names("archive", parts.length()).join(", ")}")
let joined = @zip.unzip_split(parts)
println("rejoined -> \{joined[0].name}, \{joined[0].data.length()} bytes")
}
aes-256 -> secret.txt, 14 bytes volumes -> 3 pieces: archive.z01, archive.z02, archive.zip rejoined -> secret.txt, 14 bytes
A wrong password is caught by the two-byte verification value, and a tampered payload by the tag, both before any byte is returned.
A response body arrives in chunks; a file is read four kilobytes at a time. The
Writer/Reader pair takes it a piece at a time and never
holds the whole thing — only the 32 KiB window a back-reference can reach, plus
four bytes of running checksum. push returns what is ready;
finish flushes the rest and checks the trailer.
Every codec carries the pair. What each one holds between calls is its own
unit — a block for deflate, lz4, zstd and
a brotli meta-block, a chunk for snappy and xz, a
block for bzip2 — and lzma, which has no blocks at
all, holds the dictionary a copy can reach into and carries its range coder
from token to token.
fn main raise {
let out : Array[Byte] = []
for i in 0..<4000 {
for b in "the quick brown fox jumps over the lazy dog. " {
out.push((b.to_int() % 256).to_byte())
}
out.push((i % 251).to_byte())
}
let source = Bytes::from_array(out)
println("body -> \{source.length()} bytes")
let writer = @gzip.Writer::new(level=9)
let packed : Array[Byte] = []
let mut at = 0
while at < source.length() {
let stop = if source.length() - at < 4096 { source.length() } else { at + 4096 }
for b in writer.push(source[at:stop]) { packed.push(b) }
at = stop
}
for b in writer.finish() { packed.push(b) }
let gzipped = Bytes::from_array(packed)
println("gzip -> \{gzipped.length()} bytes (\{100 - gzipped.length() * 100 / source.length()}% smaller)")
let reader = @gzip.Reader::new()
let got : Array[Byte] = []
let mut at2 = 0
while at2 < gzipped.length() {
let stop = if gzipped.length() - at2 < 777 { gzipped.length() } else { at2 + 777 }
for b in reader.push(gzipped[at2:stop]) { got.push(b) }
at2 = stop
}
for b in reader.finish() { got.push(b) }
println("gunzip -> \{Bytes::from_array(got).length()} bytes, matches: \{Bytes::from_array(got) == source}")
}
body -> 184000 bytes gzip -> 1983 bytes (99% smaller) gunzip -> 184000 bytes, matches: true
The home page runs the same algorithm in the browser; here it is in MoonBit, on a body fed back in pieces that fall inside headers and codes.
The numbers are one 184,000-byte body of English text with a periodic tail,
printed by src/examples/stream and src/examples/formats.
They are the same body the home page's
strata show — here as a table, with what each format is for.
| Format | Bytes | Of original | Reach for it when |
|---|---|---|---|
| gzip | 1,983 | 1.1% | A single file on a wire — HTTP, a log, a download. |
| zlib | 2,667 | 1.4% | The deflate HTTP content coding; a stream wrapped by someone else's framing. |
| deflate (dynamic) | 2,661 | 1.4% | Raw DEFLATE when no container is wanted at all. |
| lz4 block | 1,997 | 1.1% | Decode speed above everything — a cache, a transport. |
| lz4 frame | 4,576 | 2.5% | lz4 with a header, a content checksum and skippable frames. |
| snappy block | 9,884 | 5.4% | The framing Google uses — protobuf on the wire, Parquet on disk. |
| snappy frame | 12,439 | 6.8% | Snappy with per-chunk masked CRC-32C. |
| brotli | 682 | 0.4% | Static assets on the web; it is the codec that serves this page's text. |
| zstd | 3,042 | 1.7% | A general replacement for gzip and zlib, with a fast decoder. |
| bzip2 | 853 | 0.5% | When the setup cost is worth it on highly repetitive text. |
| xz | 468 | 0.3% | Distribution tarballs, where size matters most and decode time does not. |
| lzma alone | 415 | 0.2% | The codec xz and 7z are built on, on its own. |
| — nothing | 184,000 | 100% | The body, uncompressed, for reference. |
brotli takes a quality from 0 to 11 — higher is slower
and never larger — and the row above is quality 9, the default, which already
models literals by context, splits the command and literal streams into two block
types each when the cut pays, and codes words the static dictionary knows as
references rather than letters. Quality 10 and 11 add the one thing 9 does not
do: they choose the parse instead of taking it, walking the block for the
cheapest path to every position and keeping whichever round encoded smallest.
Pick by what the caller does with the bytes. Servicing a request: lz4 or snappy. Storing a file people download: gzip or zstd. Shipping a tarball: gzip for speed, xz for size. Every function above is documented in the API reference.