A test runner that replaces the built-in one, and benchmarks that sit beside your tests.

The test runner writes plain text to stderr. A terminal gets colored dots; a pipe or a CI log gets one line per failure and nothing for passes, which is what go test prints. The benchmark side follows Go's testing.B: a benchmark is a function that loops, and the output is the Go benchmark format, so benchstat compares two runs without any glue.

Both halves are one dependency and need Zig 0.16.

• Getting started with the runner: wire it into build.zig, then read the output.
• Getting started with benchmarks: write a suite, wire the executable, compare two runs.
• Test runner flags: formats, filtering, timeouts, flake hunting.
• Writing benchmarks: the loop, sub-benchmarks, names, and output.
• For library authors: keep the runner and benchmarks out of your consumers' builds.
• Limits: what the language does not allow this runner to do.

The API reference documents every symbol.

Getting started with the runner

Install

zig fetch --save git+https://github.com/baskingspot/testingx

Wire it into build.zig

Point .test_runner at the runner inside the dependency and turn off caching for the run step:

const testingx = b.dependency("testingx", .{ .target = target });

const tests = b.addTest(.{
    .root_module = mod,
    .test_runner = .{ .path = testingx.path("src/runner.zig"), .mode = .simple },
});
const run_tests = b.addRunArtifact(tests);
run_tests.addArg(b.fmt("--seed=0x{x}", .{b.graph.random_seed}));

mode = .simple is what skips the binary protocol the built-in runner uses. That protocol is also why writing to stdout in a test could deadlock; with the simple mode, it cannot.

The simple mode also means the run step declares no outputs, so Zig treats it as side-effecting and runs it on every build. has_side_effects is not needed here. It matters for the built-in runner's step, which is cacheable: with a fixed seed, an unchanged build can skip the run.

The runner file stays in the dependency. Do not import it from your code: a test_runner source compiles as its own module, and a file cannot belong to two modules.

Passing the build's seed through means a failing randomized test prints the seed that reproduces it. The summary always ends with (seed: 0x...).

What you get

Without a terminal, passing tests print nothing. A failure prints a block:

--- FAIL: intentional failure (0.00ms)
    error.TestUnexpectedResult
    tests/fixtures.zig:20:5: 0x... in test.intentional failure (fixtures)
        try testing.expect(1 == 2);
        ^
testingx: 1 passed, 1 failed, 1 skipped (of 3 total) in 110ms (seed: 0x0)
TESTS FAILED

On a terminal you get dots, one per test: . pass, F fail, S skip, L leak.

The summary line is stable, and scripts may parse it:

testingx: N passed, M failed, K skipped (of T total) in Xms (seed: 0x...)
ALL TESTS PASSED

Leaks and error logs add clauses to the count: , 1 leaked, , 2 error logs, , 1 flaky.

A test that leaks memory fails. A test that emits an error level log fails, even when it returns success. Both match the built-in runner.

Getting started with benchmarks

Write the suite

The suite root is a normal file with a benchmarkMain function. It registers what runs. addMod takes a module and registers every pub fn benchmarkX(b: *testingx.B) !void at its top level; add registers one function under a name you choose.

const testingx = @import("testingx");

const sort_bench = @import("sort_bench.zig");

pub fn benchmarkMain(b: *testingx.B) !void {
    try b.addMod(sort_bench);
    try b.add("Parser/Baseline", benchmarkParserBaseline);
}

fn benchmarkParserBaseline(b: *testingx.B) !void {
    var sink: usize = 0;
    defer b.keepAlive(sink);
    while (try b.loop()) sink +|= parse(input);
}

Put the import in a top-level const. An inline @import inside benchmarkMain reads worse and trips ziglint's Z028 when that rule is on.

addMod looks at top-level declarations only. It never descends into nested structs, so each file gets one line. Nested benchmarks are b.run calls inside a running benchmark.

Wire the executable

addBench builds a ReleaseFast executable around your suite and returns the run step:

const testingx = @import("testingx");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const tx = b.dependency("testingx", .{ .target = target });

    // ... define mylib_mod, the module the benchmarks measure:

    const run_bench = testingx.addBench(b, .{
        .module = tx.module("testingx"),
        .root_source_file = b.path("bench/bench.zig"),
        .imports = &.{.{ .name = "mylib", .module = mylib_mod }},
    });
    if (b.args) |args| run_bench.addArgs(args);
    b.step("bench", "Run benchmarks").dependOn(&run_bench.step);
}
`.imports` is how the suite reaches your code. The suite module is created for it, so it inherits nothing from the rest of the build; list every module the suite files import.

ReleaseFast is the default because Debug numbers are noise. Pass `.optimize = .ReleaseSafe` when a check depends on safety, and `.target` to cross-build.

For a profiling binary (`perf`, `objdump`), call `addBenchExe` instead and install the returned compile step.

## Run it

```sh
zig build bench

target: aarch64-linux-gnu
cpu: neoverse_v3
mode: ReleaseFast
BenchmarkParser/Baseline	1000	244.0 ns/op

--benchtime takes a duration or a fixed count: --benchtime=250ms, --benchtime=100x. --filter selects by name, with the same grammar as the test runner. --benchmem adds B/op and allocs/op. --jsonl prints one JSON object per record instead of text.

Two variables exist for CI policy, and a command-line flag wins: TESTINGX_BENCH_FILTER is --filter, and TESTINGX_BENCH_JSONL (1 or true) is --jsonl.

Two captures compare with benchstat:

zig build bench -- --benchmem --count=10 --benchtime=100ms > run1.txt
zig build bench -- --benchmem --count=10 --benchtime=100ms > run2.txt
benchstat run1.txt run2.txt

Keep captures on the same machine. Numbers from a laptop and a CI runner are different units.

Test runner flags

Pass flags after --:

zig build test -- --format=testname --filter=^fixtures.test.skip demonstration$

Environment

A few settings also come from the environment, so a shared CI workflow can change them without editing the repository. A command-line flag wins over the variable.

Formats

quiet prints nothing for passes and a block for failures, which is the default without a terminal. dots prints one character per test and is the default on a terminal. testname prints a line per test with icons:

✔ basic arithmetic passes (0.00ms)
∅ skip demonstration (0.00ms)
✖ intentional failure (0.00ms)

verbose prints Go's -v shape:

=== RUN basic arithmetic passes
--- PASS: basic arithmetic passes (0.00ms)
--- SKIP: skip demonstration (0.00ms)

Filters

Plain text matches anywhere in the name. ^ anchors the start, $ anchors the end, * matches any bytes, and commas separate alternatives.

Filters match the fully-qualified test name, which includes the module path: fixtures.test.skip demonstration. The unanchored form does not care. With anchors, use the full name.

Flake hunting

--rerun-fails=N reruns only the failures. A test that passes on a retry is reported as flaky and does not fail the exit code:

testingx: 0 passed, 0 failed, 0 skipped, 1 flaky (of 1 total) in 114ms (seed: 0x0)
testingx: flaky: flaky demonstration
ALL TESTS PASSED

--count=N reruns everything, for a test that fails once in twenty runs. --shuffle finds order dependence: run with --shuffle=on, and when a failure appears, rerun with --shuffle=<seed> from the summary. --timeout names the test that hung, which beats killing the process and guessing.

Writing benchmarks

The loop

A benchmark is a function that takes *testingx.B. b.loop() runs the body the right number of times; keep the result alive with defer b.keepAlive(sink) so the optimizer cannot delete the work.

fn benchmarkHash(b: *testingx.B) !void {
    var sink: u64 = 0;
    defer b.keepAlive(sink);
    while (try b.loop()) sink +%= hash(input);
    b.setBytes(@intCast(input.len)); // adds MB/s to the row
}

b.n is available for a hand-written loop (while (i < b.n) : (i += 1)) when the body cannot use b.loop(). b.allocator counts allocations. Every allocation through it shows up in B/op and allocs/op when --benchmem is on. b.reportMetric(value, "unit") adds a custom column. b.io is a per-benchmark Io.Threaded, for file or clock work.

Setup that should not be measured goes before the loop, or inside b.resetTimer() / b.stopTimer() / b.startTimer() when it interleaves.

Sub-benchmarks

b.run(name, fn) runs a benchmark inside the current one, the way Go's b.Run does. Names join with /:

pub fn benchmarkDecode(b: *testingx.B) !void {
    inline for (.{ "digits", "twain" }) |suite| {
        _ = try b.run("text=" ++ suite, Decode(suite).run);
    }
}

Use key=value parts for labeled matrices and plain parts for sizes or cases. benchstat groups by these parts, so Decode/text=digits compares across runs and Decode/text=twain stays separate.

Names drop the benchmark prefix when they register: benchmarkDecode is Decode, and the row prints BenchmarkDecode. Filters accept either spelling.

Output

Text output is the Go benchmark format. Configuration lines come first, then one result line per run:

target: aarch64-linux-gnu
cpu: neoverse_v3
mode: ReleaseFast
BenchmarkDecode/text=digits	1000	154125 ns/op	64.88 MB/s

--config key=value adds your own lines, repeatable, for anything the environment lines miss (a git revision, a governor setting). The environment lines are suppressed by --no-env.

A failed or skipped benchmark is a comment, not a result, so the result stream stays parseable:

# FAIL: BenchmarkBroken
# SKIP: BenchmarkSlow

--jsonl replaces the text with one JSON object per line, a configuration object first and then one per benchmark. Failed and skipped benchmarks are objects there.

In CI

Do not gate on benchmark numbers from CI machines. Run the benchmarks to prove they still execute:

zig build bench -- --benchtime=8x --count=1 --no-env

That runs every benchmark eight times and fails if any of them fails. Gate on the exit code.

For library authors

A library's consumers should not download the test runner or the benchmark tooling. Two pieces make that true.

Mark the dependency lazy in build.zig.zon:

.testingx = .{ .url = "git+https://github.com/baskingspot/testingx#<commit>", .hash = "...", .lazy = true },

Then put a guard between the part of the build that ships and the part that exists to develop it:

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});

    const lib_mod = b.addModule("mylib", .{
        .root_source_file = b.path("src/root.zig"),
        .target = target,
    });
    b.installArtifact(b.addLibrary(.{
        .name = "mylib",
        .root_module = lib_mod,
        .linkage = .static,
    }));

    // Everything below is for this repo only: tests, benchmarks, and the
    // tools they need. Consumers of the package never reach it.
    if (b.pkg_hash.len != 0) return;

    const testingx = b.dependency("testingx", .{ .target = target });
    // ... test and bench steps
}

b.pkg_hash is empty when the script runs as the root package and non-empty when it runs as someone else's dependency. The early return keeps the development graph out of their build: no runner, no benchmarks, no test step.

The guard is a boundary marker, not a header line. Put it after the last declaration a consumer needs. Too early and consumers lose a module or an installed artifact, with an error about a missing module. Too late and the dev dependencies stay on their fetch path. A library with no test or bench steps has nothing to guard, so it does not need the line at all.

Lazy alone is not enough. A consumer's build runs your build.zig, and if the script asks for the dependency on every evaluation, their build fetches it. The guard is what stops that.

The bench step needs both lazy pieces when the dependency is lazy: the build.zig type comes from lazyImport, and the module comes from lazyDependency.

if (b.lazyImport(@This(), "testingx")) |testingx| {
    const tx_dep = b.lazyDependency("testingx", .{}) orelse unreachable;
    const run_bench = testingx.addBench(b, .{
        .module = tx_dep.module("testingx"),
        .root_source_file = b.path("bench/bench.zig"),
        .imports = &.{.{ .name = "mylib", .module = mod }},
    });
    b.step("bench", "Run benchmarks").dependOn(&run_bench.step);
}

A path dependency in a lazy slot fails harder than it looks: when the directory is missing, the build script cannot compile, so every zig build fails before any step runs. Pin a git URL for anything shared.

Limits

Zig's compiler fixes what a test runner can do. Tests are fn () anyerror!void collected into builtin.test_functions, one process, no parameters.

• Tests run one at a time. There is no parallel test execution, because every test shares the globals in std.testing.
• There are no subtests and no way to attach a reason to a skip. error.SkipZigTest is the whole vocabulary.
• Fuzz mode needs the built-in runner, which speaks the server protocol. Use a -Dfuzz option and pass null for .test_runner when it is set.
• Filters match the fully-qualified name, not the short name in the output.
• Benchmarks are not tests. zig build test never runs them; they need their own step and a ReleaseFast build.

Watch mode is not a runner feature. zig build test --watch is the build system's, and it works with this runner.