Compression codecs in pure Zig: Snappy, DEFLATE, gzip, zlib, and Zstandard, plus a Brotli decoder.

One module, a namespace per codec, the same buffer discipline everywhere: the caller owns every buffer, the default APIs do not allocate, and every failure has a name. Streaming goes through std.Io. The encoders target the speed class — the zstd encoder is byte-identical to zstd -1 — and the zstd decoder verifies what the C library does not.

• Choose std or compress: what std.compress has in 0.16, and what this package adds. gzip, zlib, and Snappy are missing from std entirely; zstd in std is decode-only.
• One-shot encode and decode: the same shape in every codec. Encode returns the compressed length, decode returns the decoded length, and both take caller-owned buffers.
• Size the buffers: every encoder exports its exact worst case, and decode never writes past the length it returns.
• Streaming: Writer and Reader over std.Io, with one-call pumps.
• Dictionaries: zstd dictionaries both directions, the zlib preset, and why there is no trainer in here.
• Choose a codec: which format for which job, and what each one costs.
• Errors and corrupt input: closed error sets, named per failure, error.BufferTooSmall before any byte past the cap.
• Performance: what the 7-target EC2 grid measured, and the rule that local numbers are never claims.

This guide is the recipes. The API reference has the contract for every symbol. Each codec's README documents its API, error sets, and buffer sizes in the same terms.

Getting started

Install

zig fetch --save git+https://github.com/baskingspot/compress

Add the module to your build in build.zig:

const compress = b.dependency("compress", .{ .target = target, .optimize = optimize });
exe.root_module.addImport("compress", compress.module("compress"));

compress builds with Zig 0.16 only.

Import it

Bind a codec from the module, once per file:

const compress = @import("compress");
const gzip = compress.gzip;

If the file uses only one codec, import it directly:

const gzip = @import("compress").gzip;

std has a std.compress of its own. In a file that uses both, bind this module as compressx:

const compressx = @import("compress");
const zstd = compressx.zstd; // this package
const std_zstd = std.compress.zstd; // std, decode only

Read the examples

Every example in this guide is a complete file in docs/examples/. The test step compiles and runs each one, so the code in this guide builds:

just test

Codecs

flate is the kernel: gzip and zlib are containers over it, each adding its framing, its checksum, and its own error names. snappy, zstd, and brotli are self-contained.

Choose std or compress

std.compress in Zig 0.16 ships flate with both directions, zstd with decode only, and the LZMA family. It does not ship gzip, zlib, or Snappy at all, and it does not ship a Brotli decoder.

Use std when it has the direction you need and its shape fits. Use this package when you need the missing formats, a zstd encoder, or the same buffer discipline across every codec you touch. The decode paths here also check what std's zstd decoder does not: the frame checksum, the content size, and the dictionary ID, each with a named error.

One-shot encode and decode

The one-shot shape is identical in every codec:

const std = @import("std");
const compress = @import("compress");
const flate = compress.flate;
const gzip = compress.gzip;

// The one-shot shape is the same across codecs: encode returns the
// compressed length, decode returns the decoded length, and both take
// caller-owned buffers.
test "a one-shot round trip in both codecs" {
    const data = "the same buffer shape in every codec";

    var member: [64]u8 = undefined;
    const member_len = try gzip.encode.compress(data, &member, .{});

    var plain: [64]u8 = undefined;
    const plain_len = try gzip.decode.decompress(member[0..member_len], &plain);
    try std.testing.expectEqualSlices(u8, data, plain[0..plain_len]);

    var raw: [64]u8 = undefined;
    const raw_len = try flate.encode.compress(data, &raw, .{});

    var back: [64]u8 = undefined;
    const back_len = try flate.decode.decompress(raw[0..raw_len], &back, null);
    try std.testing.expectEqualSlices(u8, data, back[0..back_len]);
}

// Decode never writes past the returned length, so a bigger buffer
// keeps its tail untouched.
test "decode leaves the bytes past the decoded length alone" {
    const data = "sentinel rule";
    var member: [64]u8 = undefined;
    const member_len = try gzip.encode.compress(data, &member, .{});

    var plain: [64]u8 = undefined;
    @memset(&plain, 0xAA);
    const plain_len = try gzip.decode.decompress(member[0..member_len], &plain);
    try std.testing.expectEqualSlices(u8, data, plain[0..plain_len]);
    for (plain[plain_len..]) |b| try std.testing.expectEqual(@as(u8, 0xAA), b);
}

Encode takes source and target and returns the compressed length. Decode takes the compressed bytes and a target cap and returns the decoded length. target on the decode side is a cap, not a promise: the decode may fill the cap on its way to a refusal, but a result that does not fit fails error.BufferTooSmall and nothing past the cap is ever written. The returned length carries the other half of the rule: decode never writes past it, so a larger target keeps its tail untouched.

Size the buffers

Never guess a size. Every encoder exports its exact worst case, and Snappy's block form tells the decoder its exact decoded length up front:

const std = @import("std");
const compress = @import("compress");
const zstd = compress.zstd;
const zlib = compress.zlib;
const gzip = compress.gzip;
const snappy = compress.snappy;

// Sizing the target is the encoder's job. Each codec exports its own exact
// worst case, so no sizing is a guess.
test "the encoders size their own targets" {
    const data = "a target sized by the encoder, not by a guess";

    var member: [gzip.encode.maxCompressedLength(data.len)]u8 = undefined;
    const member_len = try gzip.encode.compress(data, &member, .{});
    try std.testing.expect(member_len <= member.len);

    var stream: [zlib.encode.maxCompressedLength(data.len)]u8 = undefined;
    const stream_len = try zlib.encode.compress(data, &stream, .{});
    try std.testing.expect(stream_len <= stream.len);

    var frame: [zstd.encode.maxCompressedLength(data.len)]u8 = undefined;
    const frame_len = try zstd.encode.compress(data, &frame, .{});
    try std.testing.expect(frame_len <= frame.len);

    // snappy's raw-block form: with the exact size, `compressBlock` cannot
    // fail, and its error set drops to `error{}` at the call site.
    var block: [snappy.encode.maxCompressedLength(data.len)]u8 = undefined;
    const block_len = try snappy.encode.compressBlock(data, &block);
    try std.testing.expect(block_len <= block.len);
}

// Sizing the decode target differs by format. Only snappy tells the caller
// the decoded length up front: the block header is a varint. For the other
// codecs the caller knows the size or bounds it.
test "snappy reads the decoded length from the block header" {
    const data = "how big is the result?";
    var block: [snappy.encode.maxCompressedLength(data.len)]u8 = undefined;
    const block_len = try snappy.encode.compressBlock(data, &block);

    try std.testing.expectEqual(
        data.len,
        try snappy.decode.decompressedBlockLength(block[0..block_len]),
    );

    // The other formats keep no header field the decoder trusts for this:
    // gzip and zlib carry the decoded size only in their trailers, and a
    // zstd frame the caller produced may name it in the header, but the
    // decode still takes a target cap and returns the decoded length.
    var frame: [zstd.encode.maxCompressedLength(data.len)]u8 = undefined;
    const frame_len = try zstd.encode.compress(data, &frame, .{});
    var plain: [64]u8 = undefined;
    const plain_len = try zstd.decode.decompress(frame[0..frame_len], &plain, null);
    try std.testing.expectEqualSlices(u8, data, plain[0..plain_len]);
}

With the exact worst case, snappy.encode.compressBlock cannot fail, and its error set drops to error{} at the call site. For the other codecs, decode sizing is the caller's: gzip and zlib carry the decoded size only in their trailers, and a decode that must know it up front needs the caller's bound. Decode never writes past the length it returns, so a larger target keeps its tail untouched — every codec's test suite pins that rule with sentinels.

Streaming

A Writer compresses into an Io.Writer, and a Reader decompresses from an Io.Reader. finish is required on the encode side: it writes the trailer. One-call pumps move a whole stream:

const std = @import("std");
const compress = @import("compress");
const gzip = compress.gzip;

// A Writer compresses into an Io.Writer, and a Reader decompresses from an
// Io.Reader. `finish` is required on the encode side: it writes the trailer.
fn streamingRoundTrip(gpa: std.mem.Allocator, input: []const u8) !void {
    var member: std.Io.Writer.Allocating = .init(gpa);
    defer member.deinit();

    var wbuf: gzip.Writer.Buffer = undefined;
    var w: gzip.Writer = .init(&member.writer, &wbuf, .{});
    try w.writer.writeAll(input);
    try w.finish();

    var in: std.Io.Reader = .fixed(member.written());
    var rbuf: gzip.Reader.Buffer = undefined;
    var r: gzip.Reader = .init(&in, &rbuf);
    const plain = try r.reader.allocRemaining(gpa, .unlimited);
    defer gpa.free(plain);
    try std.testing.expectEqualSlices(u8, input, plain);
}

test "a streaming round trip through Io" {
    try streamingRoundTrip(std.testing.allocator, "streamed through the Io interfaces");
}

// The one-call forms pump a whole input through. On the Reader side the
// gzip framing is self-delimiting, so one `streamAll` walks every member
// in a multi-member stream.
test "streamAll pumps a whole stream in one call" {
    const input = "one call, both directions";

    var member: std.Io.Writer.Allocating = .init(std.testing.allocator);
    defer member.deinit();
    var in: std.Io.Reader = .fixed(input);
    _ = try gzip.Writer.streamAll(&in, &member.writer, .{});

    var back: std.Io.Writer.Allocating = .init(std.testing.allocator);
    defer back.deinit();
    var source: std.Io.Reader = .fixed(member.written());
    _ = try gzip.Reader.streamAll(&source, &back.writer);
    try std.testing.expectEqualSlices(u8, input, back.written());
}

// The buffers are named types the caller owns. gzip's are small enough for
// a stack local; zstd's are not, and its `*Alloc` variants take an
// allocator for them.
test "the caller owns the streaming buffer" {
    var out: std.Io.Writer.Allocating = .init(std.testing.allocator);
    defer out.deinit();

    var wbuf: gzip.Writer.Buffer = undefined; // [98303]u8: a stack local is fine
    var in: std.Io.Reader = .fixed("caller-owned, stack-sized");
    var w: gzip.Writer = .init(&out.writer, &wbuf, .{});
    try w.writer.writeAll("caller-owned, stack-sized");
    try w.finish();
    _ = ∈
}

The streaming buffers are named types the caller owns. gzip's is [98303]u8 — a stack local. zstd's are not: the Reader window is 8 MiB and Writer.Buffer(19) is 1.44 MiB, so its *Alloc pumps take an allocator for exactly those buffers. Keep every streaming buffer in static or heap storage, not on the stack, unless its type is small enough to prove otherwise.

The container codecs are self-delimiting at the member boundary: gzip.Reader.streamAll walks every member in a multi-member stream, and the exact boundary stays visible when a caller needs to stop there. Snappy's streaming framing is this package's own length-prefixed form, not Google's: it has no end marker and no checksum, so a stream cut at a block boundary reads as complete. Snappy framing is a decision to make with that in mind.

Dictionaries

A dictionary is caller bytes, and both sides must supply the same bytes. zstd takes a parsed Dictionary: parse reads the formatted form that zstd --train writes, and initRaw takes any bytes at all, used as history with no entropy tables. The zlib preset is raw bytes in both directions, with FDICT and the DICTID check the RFC describes:

const std = @import("std");
const compress = @import("compress");
const zstd = compress.zstd;

// A dictionary is caller bytes. `Dictionary.parse` takes the formatted form
// that `zstd --train` writes; `initRaw` takes any bytes at all, used as
// history with no entropy tables. The value borrows the bytes: keep them
// alive as long as the dictionary is.
test "a raw dictionary round-trips both directions" {
    const dict_file = "a small dictionary of bytes the input will resemble";
    const data = "the input resembles the dictionary of bytes";

    const dict = zstd.Dictionary.initRaw(dict_file);

    var frame: [zstd.encode.maxCompressedLength(data.len)]u8 = undefined;
    const frame_len = try zstd.encode.compress(data, &frame, .{ .dict = &dict });

    var plain: [64]u8 = undefined;
    const plain_len = try zstd.decode.decompress(frame[0..frame_len], &plain, &dict);
    try std.testing.expectEqualSlices(u8, data, plain[0..plain_len]);
}

// A raw dictionary carries no ID: `initRaw` content has no magic, so the
// frame the encoder writes cannot name it. A decode without the
// dictionary proceeds until a match reaches back into history that is not
// there, and fails OffsetTooFar — the honest refusal for the raw form.
// A formatted dictionary (the `zstd --train` output `parse` reads) carries
// a DICTID: its frames fail DictionaryRequired with none supplied and
// DictionaryMismatch for the wrong one.
test "a raw dictionary frame is honest about what it needs" {
    const right = zstd.Dictionary.initRaw("the dictionary the frame was made with");
    const data = "the dictionary the frame was made with, and the data";

    var frame: [zstd.encode.maxCompressedLength(data.len)]u8 = undefined;
    const frame_len = try zstd.encode.compress(data, &frame, .{ .dict = &right });

    var plain: [128]u8 = undefined;
    try std.testing.expectError(
        error.OffsetTooFar,
        zstd.decode.decompress(frame[0..frame_len], &plain, null),
    );
}

const std = @import("std");
const compress = @import("compress");
const zlib = compress.zlib;

// The zlib preset dictionary is raw bytes (RFC 1950 §2.2 FDICT): the last
// 32 KiB are the decoder's history, and both directions must supply the
// same bytes. A dictionary longer than 32 KiB contributes only its tail.
test "a preset dictionary round-trips both directions" {
    const dict = "the same bytes to both sides of the stream";
    const data = "the same bytes appear in the data, and the matches reach back into the dictionary";

    var stream: [128]u8 = undefined;
    const stream_len = try zlib.encode.compress(data, &stream, .{ .dict = dict });

    var plain: [128]u8 = undefined;
    const plain_len = try zlib.decode.decompress(stream[0..stream_len], &plain, dict);
    try std.testing.expectEqualSlices(u8, data, plain[0..plain_len]);
}

// FDICT is part of the stream, so the decoder refuses to guess: a stream
// that says a dictionary is present fails DictionaryRequired when none is
// supplied, and the wrong dictionary fails DictionaryMismatch.
test "the wrong dictionary fails by name" {
    const dict = "the dictionary the stream was made with";
    const data = "some data that matches the dictionary";

    var stream: [128]u8 = undefined;
    const stream_len = try zlib.encode.compress(data, &stream, .{ .dict = dict });

    var plain: [128]u8 = undefined;
    try std.testing.expectError(
        error.DictionaryRequired,
        zlib.decode.decompress(stream[0..stream_len], &plain, null),
    );
    try std.testing.expectError(
        error.DictionaryMismatch,
        zlib.decode.decompress(stream[0..stream_len], &plain, "not the dictionary"),
    );
}

The value borrows the file bytes: keep them alive as long as the dictionary is. A raw dictionary carries no ID — a frame made with one decodes without it only until a match reaches back into history that is not there, and fails error.OffsetTooFar. A formatted dictionary carries a DICTID, and its frames refuse by name: DictionaryRequired when none is supplied, DictionaryMismatch for the wrong one. The zlib preset fails the same way. gzip has no dictionary concept (RFC 1952 defines none), and Brotli's static dictionary already ships inside the decoder.

There is no trainer in this package, on purpose. Training is a tooling decision the zstd CLI already makes well; this package is the consumer side.

Choose a codec

Pick the format for the job, not the fastest name:

The zstd encoder is byte-identical to zstd -1, which pins it to the fast class: .fast is level 1, and .ratio is a reserved seat that returns error.Unimplemented until a higher level exists. The DEFLATE encoder writes fixed-Huffman and stored blocks only, which is the speed class of that family; gzip and zlib inherit it.

Errors and corrupt input

Every failure has a name, the sets are closed per codec, and the names say what was wrong with the bytes rather than which layer noticed:

const std = @import("std");
const compress = @import("compress");
const flate = compress.flate;
const gzip = compress.gzip;
const zstd = compress.zstd;

// Every failure has a name. The error sets are closed per codec, so a
// switch can be exhaustive, and the names say what was wrong with the
// bytes rather than which layer noticed.
test "corrupt input fails with a named error" {
    // A gzip header with the wrong magic: ID1, ID2, or CM wrong (§2.3.1).
    const not_gzip = "not a gzip member at all, just text";
    var plain: [64]u8 = undefined;
    try std.testing.expectError(error.BadHeader, gzip.decode.decompress(not_gzip, &plain));

    // A raw-deflate stream cut before its final block: the input ended
    // early, and the error says so.
    const data = "a stream that will be cut short";
    var raw: [64]u8 = undefined;
    const raw_len = try flate.encode.compress(data, &raw, .{});
    try std.testing.expectError(
        error.Truncated,
        flate.decode.decompress(raw[0 .. raw_len - 2], &plain, null),
    );

    // A zstd frame without zstd's magic: the first four bytes are the
    // frame's whole identity (§3.1.1.1).
    try std.testing.expectError(error.BadMagic, zstd.decode.decompress("no magic here", &plain, null));
}

// Decode treats the target as a cap, not a promise: a result that does
// not fit returns error.BufferTooSmall. The decode may fill the cap on
// its way to the refusal — the guarantee is that nothing past the cap
// is ever written.
test "a target too small fails before writing past it" {
    const data = "a decoded stream that will not fit";
    var member: [64]u8 = undefined;
    const member_len = try gzip.encode.compress(data, &member, .{});

    var tiny: [4]u8 = @splat(0xAA);
    try std.testing.expectError(
        error.BufferTooSmall,
        gzip.decode.decompress(member[0..member_len], &tiny),
    );
}

// The returned length is the other half of the rule: decode never writes
// past the length it returns, so a larger target keeps its tail untouched.
// Every codec's test suite pins this with sentinels.

Hostile input gets error returns, never assertions: the decoders treat every input as untrusted, and the fuzz suites pin the error names per mutation class. Truncated input fails by name too — error.Truncated for a DEFLATE stream cut before its final block, and each container adds its own names for its framing (error.BadHeader for gzip's magic, error.BadMagic for zstd's).

Performance

The encoders and decoders target the fast class, and the numbers are measured, not asserted. Every comparison runs on a 7-target EC2 grid — ARM and Intel — against the C reference libraries, 5 rounds each with an A/A control, and every claim names a run and a results file in docs/results/:

• The zstd encoder is at parity to faster than facebook/zstd's C library at the level-1 class on literal-heavy input on every ARM target, with both sides paying the frame checksum. It is 4-17x faster than klauspost/compress's Go zstd on entropy-heavy input. The zstd decoder verifies the frame checksum and the content size that the C library skips.
• The Brotli decoder is 2-3x faster than google/brotli's C decoder on copy-structure input, and the remaining gaps are entropy rows with ranked, measured causes.
• Local just bench numbers are never claims. The grid and the methodology are in bench/README.md.

No allocation on the hot paths, no Io value held, and the caller's buffers are the only memory the codecs touch.

Framing, and what is not implemented

• A Brotli encoder, and Brotli custom dictionaries.
• Level.ratio in flate and zstd: a reserved seat that returns error.Unimplemented.
• A Zstandard dictionary trainer.

Everything else a caller needs is on the API reference page.