utf API reference Changelog View on GitHub

SIMD Unicode validation and transcoding, plus base64, in Zig with no C.

simdutf is the C++ library for this job, and it earned its place. This package does the same job in Zig: validate UTF-8, ASCII, UTF-16LE/BE, and UTF-32; convert between any two of UTF-8, UTF-16LE/BE, UTF-32, and Latin-1; size the output; and encode or decode base64. Nothing allocates. Every function takes the buffers it needs, and a length function says how big the output must be.

On a generic x86-64 build the AVX-512, AVX2, and SSE kernels are picked at run time. A native or named CPU picks at compile time. On aarch64 the NEON kernels are picked at compile time.

The API reference documents every public symbol.

Getting started

Install

zig fetch --save git+https://github.com/baskingspot/utf

Add the module in build.zig:

const utf = b.dependency("utf", .{ .target = target, .optimize = optimize });
exe.root_module.addImport("utf", utf.module("utf"));

utf builds with Zig 0.16 only.

Import it

const utf = @import("utf");

utf does not collide with a std namespace. std.unicode keeps its name.

Read the examples

docs/examples/root.zig is the code this guide shows, compiled against the public module and run by just test.

How it picks a kernel

A build for a generic x86-64 CPU (-Dcpu=baseline, x86_64_v2, x86_64_v3, or a cross target with no CPU given) compiles the package once per ISA tier above the target and picks the best one on the first call with cpuid and XCR0. A native or named CPU selects at compile time instead. -Ddispatch=false, or .dispatch = false in the dependency's arguments, turns dispatch off.

On aarch64, NEON has no runtime choice to make.

Validating text

One validator per encoding. Each takes a slice and returns a bool.

FunctionInputAccepts
validate.utf8[]const u8well-formed UTF-8
validate.ascii[]const u8every byte below 0x80
validate.utf16le[]const u16well-formed UTF-16 in little-endian bytes
validate.utf16be[]const u16well-formed UTF-16 in big-endian bytes
validate.utf32[]const u32Unicode scalar values in native order

Ask whether it is valid

if (!utf.validate.utf8(input)) return error.InvalidUtf8;

The answer is the scalar validator's, in utf.scalar. Every SIMD path is tested against it, error kind and position included.

Ask why it failed

The WithErrors form returns a Result instead of a bool.

const result = utf.validate.utf8WithErrors(input);
if (result.err != .none) {
    std.log.err("UTF-8 error {s} at byte {d}", .{ @tagName(result.err), result.count });
}

On success count is the number of code units validated. On failure it is the position of the first bad one:

Result is a plain struct, not an error union, so it is checked with if, not try or catch.

The error kinds

The base64 error kinds are in Base64.

UTF-16 byte order is the bytes, not the values

[]const u16 is a view of memory, and utf16le and utf16be name the order those bytes are in, as simdutf's char16_t functions do. On a little-endian machine, a big-endian stream read into a byte buffer is a []const u16 with every value swapped, and utf16be swaps it back. Do not decode the units to native values first: that throws away the order the name is asking about.

UTF-32 is native order. There is no utf32be.

Transcoding

convert has two forms for every pair of encodings.

The names read source, then destination. Every pair among UTF-8, UTF-16LE, UTF-16BE, UTF-32, and Latin-1 is there, except Latin-1 to itself.

Size the output first

Every conversion needs a destination buffer. The length functions say how big it must be:

const source = "Héllo, 世界";
const units = utf.length.utf16FromUtf8(source);

var buffer: [64]u16 = undefined;
const result = utf.convert.utf8ToUtf16le(source, buffer[0..units]);
if (result.err != .none) return error.InvalidUtf8;

On success result.count is the number of code units written. A buffer sized by the matching length function always fits, even when the input turns out to be invalid. If the destination is smaller, the checked form fails with output_buffer_too_small at the first character that does not fit, and the destination holds everything before it.

The checked form names what failed

const units = [_]u16{ 0x41, 0xD800, 0x42 };
var buffer: [8]u8 = undefined;

const result = utf.convert.utf16leToUtf8(&units, &buffer);
// result.err is .surrogate, and result.count is 1.

The error kinds and positions are the validators' own. One difference: a conversion can also fail because the output ran out.

The valid form skips the checks

Use it when the input already passed a validator, or came from a source that guarantees it. It returns a usize and never fails.

const written = utf.convert.validUtf8ToUtf16le(source, buffer[0..units]);

It assumes the destination is large enough. Size it with the length function.

Latin-1 is 256 characters

Every byte is a code point from U+0000 to U+00FF. Conversion from Latin-1 cannot fail except on the output size. Conversion to Latin-1 fails with too_large for any character above U+00FF, including every emoji.

It is ISO-8859-1, not Windows-1252: bytes 0x80 to 0x9F become control characters, not quotation marks or dashes.

Byte order

utf16le and utf16be work the same way here as in the validators. Pass the bytes as []u16 and pick the name for the order they are in. UTF-32 is native order, and a big-endian UTF-32 stream needs its values swapped before it gets here.

Counting and sizing

length answers two questions without writing anything: how many code units a conversion needs, and how many code points an input holds.

Size a conversion

The name is destination, then source:

const units = utf.length.utf16FromUtf8(source);

Every function is defined on any input, valid or not, and returns exactly what the scalar reference returns. That is why a buffer sized by one always fits the matching checked conversion.

Count code points

const points = utf.length.countUtf8("aé日");         // 3
const units = [_]u16{ 0x61, 0xD83D, 0xDE00, 0x7A };
const same = utf.length.countUtf16le(&units);       // 3, because the pair is one

countUtf8, countUtf16le, and countUtf16be count code points, so a surrogate pair counts once. countUtf32 does not exist: each u32 is one code point.

Base64

utf.base64 encodes and decodes with the standard alphabet (+ and /) or the URL alphabet (- and _).

Encode

var buffer: [16]u8 = undefined;
const characters = utf.base64.encode(&buffer, "hello", .{});
// buffer[0..characters] is "aGVsbG8=".

encodedLen(n, options) gives the buffer size for n bytes. EncodeOptions has two fields:

const characters = utf.base64.encode(&buffer, "\xFB\xFF", .{
    .alphabet = .url,
    .padding = false,
});
// "-_8"

Decode

var buffer: [8]u8 = undefined;
const result = utf.base64.decode(&buffer, "aGVsbG8=", .{});
if (result.err != .none) return error.InvalidBase64;
// buffer[0..result.count] is "hello".

Size the destination with maxDecodedLen(n), the most bytes n input characters can produce. On success count is the number of bytes written.

DecodeOptions has two fields: alphabet and last_chunk.

Whitespace is skipped

decode skips tab, line feed, form feed, carriage return, and space, wherever they appear. That matches WHATWG's forgiving base64 and simdutf's behavior, so a MIME or PEM body wrapped at 76 columns decodes without preprocessing.

No option rejects whitespace. A decoder that must reject it needs its own scan first.

The last chunk

last_chunk decides what happens to two or three characters at the end that are not padded to four:

const strict = utf.base64.decode(&buffer, "aGVsbG8", .{ .last_chunk = .strict });
// strict.err is .base64_input_remainder, and strict.count is 7.

What a failure reports

On failure the contents of the destination are unspecified.

Decode in place

The destination may start where the source does:

var buffer: [8]u8 = "aGVsbG8=".*;
const result = utf.base64.decode(&buffer, buffer[0..], .{});

The result is the same as decoding into a separate buffer. On failure the source is unspecified too. Otherwise the two buffers must not overlap. There is no in-place encode.

Limits