utf API reference Changelog View on GitHub

API reference

SIMD Unicode validation and transcoding, plus base64, in Zig with no C.

enum Error

src 
pub const Error = result.Error

Why an input was rejected. The UTF kinds and their positions match simdutf's error_code and result, so simdutf is a differential oracle.

Fields

none none
header_bits header_bits

A UTF-8 byte of the form 11111xxx.

too_short too_short

A UTF-8 sequence that ends before its continuation bytes do.

too_long too_long

A continuation byte where none was expected.

overlong overlong

A UTF-8 sequence longer than the shortest one for its code point, including the leads 0xC0 and 0xC1.

too_large too_large

A code point above U+10FFFF, including 4-byte sequences led by 0xF5-0xF7.

surrogate surrogate

A surrogate code point (U+D800-U+DFFF) in UTF-8 or UTF-32, or an unpaired surrogate in UTF-16.

not_ascii not_ascii

A byte above 0x7F where only ASCII is allowed.

invalid_base64_character invalid_base64_character

A byte that is not in the base64 alphabet and not ignorable space.

base64_input_remainder base64_input_remainder

Base64 input whose last chunk cannot be decoded under the options.

base64_extra_bits base64_extra_bits

Base64 input with nonzero bits in the padding position.

output_buffer_too_small output_buffer_too_small

The output buffer is smaller than the operation needs.

struct Result

src 
pub const Result = result.Result

Fields

err: Error
count: usize

Functions

pub fn ok(count: usize) Result

src

pub fn fail(err: Error, position: usize) Result

src

module scalar

src

Scalar reference implementations, one code unit at a time. They define the expected result for every SIMD path: a SIMD function must return exactly what its scalar counterpart returns, error kind and position included, on every input.

Functions

pub fn validateUtf8(bytes: []const u8) Result

src

Validates UTF-8, classifying errors exactly as simdutf's scalar validate_with_errors does, so simdutf can serve as a differential oracle. On failure count is the index of the first byte of the ill-formed sequence.

pub fn validateAscii(bytes: []const u8) Result

src

Validates ASCII. On failure count is the index of the first byte above 0x7F.

pub fn validateUtf16le(units: []const u16) Result

src

Validates UTF-16 stored little-endian. On failure count is the index of the unpaired surrogate: a low one with no high one before it, or a high one not followed by a low one.

pub fn validateUtf16be(units: []const u16) Result

src

Validates UTF-16 stored big-endian, as validateUtf16le does.

pub fn validateUtf32(units: []const u32) Result

src

Validates UTF-32 in native byte order. On failure count is the index of the first value above U+10FFFF or in the surrogate range.

module scalar_convert

src

Scalar reference transcoders and length functions, one code point at a time. They define the result of every conversion: each SIMD path in convert.zig and length.zig returns exactly what its counterpart here returns, error kind and position included, on every input.

more

A checked conversion (utf8ToUtf16, ...) validates as it converts and stops at the first ill-formed character, with simdutf's error kind and the index of that character in the input. Before writing a character it checks that the output has room for it, and fails with output_buffer_too_small at that character's index when it does not; the output then holds the conversion of everything before it. A buffer sized by the matching length function never fails that way, even on invalid input, because the length functions count every prefix at least as large as it converts.

A valid conversion (validUtf8ToUtf16, ...) assumes well-formed input and an output buffer the matching length function sized; it returns the number of code units written.

UTF-16 functions take the byte order of the code units in memory as a comptime Endian, as simdutf's char16_t functions do. UTF-32 is native.

Functions

pub inline fn swap16(comptime e: Endian, unit: u16) u16

src

A UTF-16 code unit stored in byte order e, as a native value, or a native value as one stored in byte order e: the same swap both ways.

pub fn utf8ToUtf16(comptime e: Endian, src: []const u8, dst: []u16) Result

src

pub fn validUtf8ToUtf16(comptime e: Endian, src: []const u8, dst: []u16) usize

src

pub fn utf8ToUtf32(src: []const u8, dst: []u32) Result

src

pub fn validUtf8ToUtf32(src: []const u8, dst: []u32) usize

src

pub fn utf8ToLatin1(src: []const u8, dst: []u8) Result

src

Converts UTF-8 holding only U+0000-U+00FF to Latin-1, classifying errors as simdutf does: a two-byte character above U+00FF, and any three- or four-byte lead, is too_large at its first byte, before its continuation bytes are looked at.

pub fn validUtf8ToLatin1(src: []const u8, dst: []u8) usize

src

pub fn utf16ToUtf8(comptime e: Endian, src: []const u16, dst: []u8) Result

src

pub fn validUtf16ToUtf8(comptime e: Endian, src: []const u16, dst: []u8) usize

src

pub fn utf16ToUtf32(comptime e: Endian, src: []const u16, dst: []u32) Result

src

pub fn validUtf16ToUtf32(comptime e: Endian, src: []const u16, dst: []u32) usize

src

pub fn utf16ToLatin1(comptime e: Endian, src: []const u16, dst: []u8) Result

src

Every code unit above 0xFF, surrogates included, is too_large at its index, as simdutf reports it.

pub fn validUtf16ToLatin1(comptime e: Endian, src: []const u16, dst: []u8) usize

src

pub fn utf16ToUtf16Swapped(comptime e: Endian, src: []const u16, dst: []u16) Result

src

Converts UTF-16 in byte order e to the other byte order, failing on an unpaired surrogate as utf16ToUtf8 does.

pub fn validUtf16ToUtf16Swapped(src: []const u16, dst: []u16) usize

src

pub fn utf32ToUtf8(src: []const u32, dst: []u8) Result

src

pub fn validUtf32ToUtf8(src: []const u32, dst: []u8) usize

src

pub fn utf32ToUtf16(comptime e: Endian, src: []const u32, dst: []u16) Result

src

pub fn validUtf32ToUtf16(comptime e: Endian, src: []const u32, dst: []u16) usize

src

pub fn utf32ToLatin1(src: []const u32, dst: []u8) Result

src

Every code point above 0xFF is too_large at its index.

pub fn validUtf32ToLatin1(src: []const u32, dst: []u8) usize

src

pub fn latin1ToUtf8(src: []const u8, dst: []u8) Result

src

pub fn validLatin1ToUtf8(src: []const u8, dst: []u8) usize

src

pub fn latin1ToUtf16(comptime e: Endian, src: []const u8, dst: []u16) Result

src

pub fn validLatin1ToUtf16(comptime e: Endian, src: []const u8, dst: []u16) usize

src

pub fn latin1ToUtf32(src: []const u8, dst: []u32) Result

src

pub fn validLatin1ToUtf32(src: []const u8, dst: []u32) usize

src

pub fn countUtf8(src: []const u8) usize

src

Code points in UTF-8: the bytes that are not continuation bytes.

pub fn countUtf16(comptime e: Endian, src: []const u16) usize

src

Code points in UTF-16: the code units that are not low surrogates.

pub fn utf16FromUtf8(src: []const u8) usize

src

UTF-16 code units for UTF-8: one per character, two per four-byte one.

pub fn utf8FromUtf16(comptime e: Endian, src: []const u16) usize

src

UTF-8 bytes for UTF-16: every surrogate counts two, so a pair counts four. On unpaired surrogates simdutf disagrees with itself (its SIMD blocks count two, its scalar tail three); this is the per-unit rule.

pub fn utf8FromUtf32(src: []const u32) usize

src

pub fn utf16FromUtf32(src: []const u32) usize

src

pub fn utf8FromLatin1(src: []const u8) usize

src

module convert

src

Transcoding between UTF-8, UTF-16LE, UTF-16BE, UTF-32, and Latin-1, with SIMD. Callers own every buffer; nothing allocates.

more

Each conversion has two forms. The checked form (utf8ToUtf16le) validates as it converts and returns a Result: on success the number of code units written, on failure simdutf's error kind and the index in the input of the first ill-formed character. When the output runs out it fails with output_buffer_too_small at the first character that does not fit; a buffer sized by the matching length function always fits. The valid form (validUtf8ToUtf16le) assumes well-formed input and a buffer the length function sized, and returns the number of code units written. Results equal those of scalar_convert.zig on every input.

UTF-16LE and UTF-16BE name the byte order of the code units in memory, as simdutf's char16_t functions do; UTF-32 is native.

The kernels follow Lemire and Muła, "Transcoding Billions of Unicode Characters per Second with SIMD Instructions" (2022), and on AVX-512 with VBMI2 Clausecker and Lemire, "Transcoding Unicode Characters with AVX-512 Instructions" (2023). UTF-8 is decoded a 64-byte block at a time: each byte gets the value of a character ending there, and the values that end characters are packed by shuffle tables keyed by the end-of-character mask (by lane compression on AVX-512), with 12-byte windows for the tail. UTF-16 and UTF-32 are encoded by expanding code points into lanes and packing them, with ASCII and two-byte fast paths. UTF-8 is validated in the same pass with Keiser and Lemire's three-lookup check. Every table is built at comptime.

Functions

pub fn utf8ToUtf16le(src: []const u8, dst: []u16) Result

src

pub fn utf8ToUtf16be(src: []const u8, dst: []u16) Result

src

pub fn utf8ToUtf32(src: []const u8, dst: []u32) Result

src

pub fn utf8ToLatin1(src: []const u8, dst: []u8) Result

src

pub fn validUtf8ToUtf16le(src: []const u8, dst: []u16) usize

src

pub fn validUtf8ToUtf16be(src: []const u8, dst: []u16) usize

src

pub fn validUtf8ToUtf32(src: []const u8, dst: []u32) usize

src

pub fn validUtf8ToLatin1(src: []const u8, dst: []u8) usize

src

pub fn utf16leToUtf8(src: []const u16, dst: []u8) Result

src

pub fn utf16beToUtf8(src: []const u16, dst: []u8) Result

src

pub fn utf16leToUtf32(src: []const u16, dst: []u32) Result

src

pub fn utf16beToUtf32(src: []const u16, dst: []u32) Result

src

pub fn utf16leToLatin1(src: []const u16, dst: []u8) Result

src

pub fn utf16beToLatin1(src: []const u16, dst: []u8) Result

src

pub fn utf16leToUtf16be(src: []const u16, dst: []u16) Result

src

Swaps the byte order, failing on an unpaired surrogate.

pub fn utf16beToUtf16le(src: []const u16, dst: []u16) Result

src

Swaps the byte order, failing on an unpaired surrogate.

pub fn validUtf16leToUtf8(src: []const u16, dst: []u8) usize

src

pub fn validUtf16beToUtf8(src: []const u16, dst: []u8) usize

src

pub fn validUtf16leToUtf32(src: []const u16, dst: []u32) usize

src

pub fn validUtf16beToUtf32(src: []const u16, dst: []u32) usize

src

pub fn validUtf16leToLatin1(src: []const u16, dst: []u8) usize

src

pub fn validUtf16beToLatin1(src: []const u16, dst: []u8) usize

src

pub fn validUtf16leToUtf16be(src: []const u16, dst: []u16) usize

src

pub fn validUtf16beToUtf16le(src: []const u16, dst: []u16) usize

src

pub fn utf32ToUtf8(src: []const u32, dst: []u8) Result

src

pub fn utf32ToUtf16le(src: []const u32, dst: []u16) Result

src

pub fn utf32ToUtf16be(src: []const u32, dst: []u16) Result

src

pub fn utf32ToLatin1(src: []const u32, dst: []u8) Result

src

pub fn validUtf32ToUtf8(src: []const u32, dst: []u8) usize

src

pub fn validUtf32ToUtf16le(src: []const u32, dst: []u16) usize

src

pub fn validUtf32ToUtf16be(src: []const u32, dst: []u16) usize

src

pub fn validUtf32ToLatin1(src: []const u32, dst: []u8) usize

src

pub fn latin1ToUtf8(src: []const u8, dst: []u8) Result

src

pub fn latin1ToUtf16le(src: []const u8, dst: []u16) Result

src

pub fn latin1ToUtf16be(src: []const u8, dst: []u16) Result

src

pub fn latin1ToUtf32(src: []const u8, dst: []u32) Result

src

pub fn validLatin1ToUtf8(src: []const u8, dst: []u8) usize

src

pub fn validLatin1ToUtf16le(src: []const u8, dst: []u16) usize

src

pub fn validLatin1ToUtf16be(src: []const u8, dst: []u16) usize

src

pub fn validLatin1ToUtf32(src: []const u8, dst: []u32) usize

src

module length

src

How many code units each conversion writes, and how many code points an input holds. Each function is defined on any input, valid or not, and returns exactly what its counterpart in scalar_convert.zig returns; on valid input that is simdutf's answer. An output buffer of the size a length function gives always fits the matching checked conversion, even when the input turns out to be invalid.

Functions

pub fn countUtf8(src: []const u8) usize

src

Code points in UTF-8: the bytes that are not continuation bytes.

pub fn countUtf16le(src: []const u16) usize

src

Code points in UTF-16LE: the code units that are not low surrogates.

pub fn countUtf16be(src: []const u16) usize

src

Code points in UTF-16BE: the code units that are not low surrogates.

pub fn utf16FromUtf8(src: []const u8) usize

src

UTF-16 code units for UTF-8: one per character, two per four-byte one.

pub fn utf16FromUtf32(src: []const u32) usize

src

UTF-16 code units for UTF-32: one per code point, two above U+FFFF.

pub fn utf16FromLatin1(src: []const u8) usize

src

UTF-16 code units for Latin-1: one per byte.

pub fn utf32FromUtf8(src: []const u8) usize

src

UTF-32 code units for UTF-8: one per character.

pub fn utf32FromUtf16le(src: []const u16) usize

src

UTF-32 code units for UTF-16LE: one per code point.

pub fn utf32FromUtf16be(src: []const u16) usize

src

UTF-32 code units for UTF-16BE: one per code point.

pub fn utf32FromLatin1(src: []const u8) usize

src

UTF-32 code units for Latin-1: one per byte.

pub fn utf8FromUtf16le(src: []const u16) usize

src

UTF-8 bytes for UTF-16LE: a surrogate counts two, so a pair counts four.

pub fn utf8FromUtf16be(src: []const u16) usize

src

UTF-8 bytes for UTF-16BE: a surrogate counts two, so a pair counts four.

pub fn utf8FromUtf32(src: []const u32) usize

src

UTF-8 bytes for UTF-32: one to four per code point.

pub fn utf8FromLatin1(src: []const u8) usize

src

UTF-8 bytes for Latin-1: one per byte below 0x80, two above.

pub fn latin1FromUtf8(src: []const u8) usize

src

Latin-1 bytes for UTF-8: one per character.

pub fn latin1FromUtf16(src: []const u16) usize

src

Latin-1 bytes for UTF-16: one per code unit.

pub fn latin1FromUtf32(src: []const u32) usize

src

Latin-1 bytes for UTF-32: one per code point.

module base64

src

Base64 encoding and decoding with SIMD, after Muła and Lemire, "Faster Base64 Encoding and Decoding Using AVX2 Instructions" (2018), and "Base64 encoding and decoding at almost the speed of a memory copy" (2020) for AVX-512 VBMI. Each step of the main loops turns 48 bytes into 64 characters or back; the last step works on a zero-padded block and writes only what the input fills, so short inputs take the same vector code. Decoding skips ASCII whitespace as WHATWG's forgiving-base64 does: a block with whitespace has it compressed out into a staging buffer, which is decoded once it holds 64 characters. The scalar reference in scalar_base64.zig takes over at errors and at an output buffer too small to finish in vectors, and every path here returns exactly what it returns.

Constants

pub const native: Isa = …

src
value
for ([_]Isa{ .avx512vbmi, .avx2, .ssse3, .neon }) |isa| {
    if (isa.supported()) break isa;
} else .generic

The kernel encode and decode use: the widest the target supports, or with runtime dispatch, the widest the tier they run in supports.

Functions

pub fn encodedLen(n: usize, options: EncodeOptions) usize

src

The number of characters encode writes for n bytes.

pub fn maxDecodedLen(n: usize) usize

src

The most bytes decode can write for n input characters, whatever they are.

pub fn encode(dst: []u8, src: []const u8, options: EncodeOptions) usize

src

Encodes src into dst and returns the number of characters written, encodedLen(src.len, options). dst must hold that many.

pub fn decode(dst: []u8, src: []const u8, options: DecodeOptions) Result

src

Decodes base64 src into dst, skipping ASCII whitespace (tab, LF, FF, CR, and space), with padding optional unless options.last_chunk is .strict.

more

On success count is the number of bytes written. On failure it is a position in src: for invalid_base64_character, the offending character, which includes a misplaced =; for base64_input_remainder and base64_extra_bits, the end of the encoded data; for output_buffer_too_small, the first character of the chunk that does not fit, where decoding can resume with more room. A dst of maxDecodedLen(src.len) bytes always fits. On failure the contents of dst are unspecified.

dst may start where src does, to decode in place, with the same result as decoding into a buffer of its own; on failure the contents of src are then unspecified too. Otherwise the two must not overlap.

pub fn encodeWith(comptime isa: Isa, comptime alphabet: Alphabet, dst: []u8, src: []const u8, padding: bool) usize

src

pub fn decodeWith( comptime isa: Isa, comptime alphabet: Alphabet, dst: []u8, src: []const u8, last_chunk: LastChunk, ) Result

src

Types

module scalar

src

Scalar base64, one character at a time: the reference every SIMD path in base64.zig must match, result and error position included. Decoding follows WHATWG forgiving-base64 and the TC39 Uint8Array.fromBase64 last-chunk rules, with simdutf's base64_to_binary error kinds and positions.

Constants

pub const space: u8 = 0x40

src

A character's 6-bit value, or space or invalid.

pub const invalid: u8 = 0x80

src

Functions

pub fn encodedLen(n: usize, options: EncodeOptions) usize

src

The number of characters encode writes for n bytes.

pub fn maxDecodedLen(n: usize) usize

src

The most bytes decode can write for n input characters, whatever they are.

pub fn encode(dst: []u8, src: []const u8, options: EncodeOptions) usize

src

Encodes src into dst and returns the number of characters written, encodedLen(src.len, options). dst must hold that many.

pub inline fn encodeChars(dst: []u8, src: []const u8, table: *const [64]u8, padding: bool) usize

src

encode with the alphabet's characters and the padding flag apart: SIMD paths inline it, and LLVM keeps an options struct inlined into them in memory.

pub fn decodeTable(alphabet: Alphabet) *const [256]u8

src

The decoding table: each byte's value, space for the ASCII whitespace WHATWG's forgiving-base64 skips (tab, LF, FF, CR, space), and invalid for everything else, = included.

pub fn isSpace(c: u8) bool

src

pub fn trim(src: []const u8) Trim

src

pub fn decode(dst: []u8, src: []const u8, options: DecodeOptions) Result

src

Decodes base64 src into dst, skipping ASCII whitespace.

more

On success count is the number of bytes written. On failure it is a position in src: for invalid_base64_character, the offending character, which includes a misplaced = and the first of trailing padding that does not fit the last chunk; for base64_input_remainder and base64_extra_bits, the end of the encoded data, before any trailing padding and whitespace; for output_buffer_too_small, the first character of the chunk that does not fit, where decoding can resume. dst needs at most maxDecodedLen(src.len) bytes. On failure the contents of dst are unspecified.

pub fn decodeFrom(dst: []u8, src: []const u8, t: Trim, start: usize, out: usize, options: DecodeOptions) Result

src

Decodes src[i..t.len] into dst[o..] and finishes the input, where t is trim(src) and i starts a chunk: the characters before it, less whitespace, are a multiple of four and already decoded into dst[0..o]. SIMD paths call it for the part of the input they leave.

Types

enum Alphabet

src

Fields

standard standard

RFC 4648 section 4: + and /.

url url

RFC 4648 section 5: - and _.

Functions

pub fn chars(alphabet: Alphabet) *const [64]u8

src

enum LastChunk

src

What decoding does with a last chunk of two or three characters, as the TC39 lastChunkHandling option does.

Fields

loose loose

Decode it, padded or not, and ignore nonzero bits past the last byte.

strict strict

Require it to be padded to four characters with =, and reject nonzero bits past the last byte.

stop_before_partial stop_before_partial

Leave an unpadded partial chunk undecoded, without an error.

struct EncodeOptions

src

Fields

alphabet: Alphabet = .standard
padding: bool = true

Pad the output with = to a multiple of four characters.

struct DecodeOptions

src

Fields

alphabet: Alphabet = .standard
last_chunk: LastChunk = .loose

struct Trim

src

Where the encoded data ends: the input with trailing whitespace and up to two trailing = (with whitespace between them) cut off, as simdutf cuts them. A = left inside src[0..len] is an invalid character. pad is the index of the first = cut off, or len without padding.

Fields

len: usize
pad: usize
padding: u2

enum Alphabet

src 
pub const Alphabet = scalar.Alphabet

Fields

standard standard

RFC 4648 section 4: + and /.

url url

RFC 4648 section 5: - and _.

Functions

pub fn chars(alphabet: Alphabet) *const [64]u8

src

enum LastChunk

src 
pub const LastChunk = scalar.LastChunk

What decoding does with a last chunk of two or three characters, as the TC39 lastChunkHandling option does.

Fields

loose loose

Decode it, padded or not, and ignore nonzero bits past the last byte.

strict strict

Require it to be padded to four characters with =, and reject nonzero bits past the last byte.

stop_before_partial stop_before_partial

Leave an unpadded partial chunk undecoded, without an error.

struct EncodeOptions

src 
pub const EncodeOptions = scalar.EncodeOptions

Fields

alphabet: Alphabet = .standard
padding: bool = true

Pad the output with = to a multiple of four characters.

struct DecodeOptions

src 
pub const DecodeOptions = scalar.DecodeOptions

Fields

alphabet: Alphabet = .standard
last_chunk: LastChunk = .loose

enum Isa

src

The instruction sets with a kernel here. generic is plain @Vector code for any target; the others call LLVM intrinsics for byte lookups, permutes, and multiply-adds, so they need the LLVM backend.

Fields

generic generic
ssse3 ssse3
avx2 avx2
avx512vbmi avx512vbmi

AVX-512 with VBMI (vpermb, vpermi2b, vpmultishiftqb); with VBMI2 it also compresses whitespace out with vpcompressb.

neon neon

Functions

pub fn supported(isa: Isa) bool

src

Whether this build can run the kernel.

module validate

src

Validation of UTF-8, ASCII, UTF-16LE/BE, and UTF-32, with SIMD.

more

Each function checks 64-byte blocks and stops at the first block that fails. Inputs of four blocks or more check their first block unaligned and the rest from the next 64-byte boundary, so no load splits a cache line; a final partial block is the last 64 bytes of the input, overlapping the block before it. Inputs shorter than a block are zero-padded into one. The WithErrors forms find the position in the failing block: ASCII, UTF-16, and UTF-32 from its lane masks, UTF-8 by rerunning the scalar validator in scalar.zig from the last character boundary before it, so error kinds and positions are exactly the scalar validators'.

Functions

pub fn utf8(bytes: []const u8) bool

src

Reports whether bytes is well-formed UTF-8.

pub fn utf8WithErrors(bytes: []const u8) Result

src

Validates UTF-8. On failure count is the index of the first byte of the ill-formed sequence, and err says what is wrong with it.

pub fn ascii(bytes: []const u8) bool

src

Reports whether every byte is below 0x80.

pub fn asciiWithErrors(bytes: []const u8) Result

src

Validates ASCII. On failure count is the index of the first byte above 0x7F.

pub fn utf16le(units: []const u16) bool

src

Reports whether units is well-formed UTF-16 stored little-endian.

pub fn utf16leWithErrors(units: []const u16) Result

src

Validates UTF-16 stored little-endian. On failure count is the index of the unpaired surrogate.

pub fn utf16be(units: []const u16) bool

src

Reports whether units is well-formed UTF-16 stored big-endian.

pub fn utf16beWithErrors(units: []const u16) Result

src

Validates UTF-16 stored big-endian. On failure count is the index of the unpaired surrogate.

pub fn utf32(units: []const u32) bool

src

Reports whether every value is a Unicode scalar value: at most U+10FFFF and not a surrogate.

pub fn utf32WithErrors(units: []const u32) Result

src

Validates UTF-32 in native byte order. On failure count is the index of the first value above U+10FFFF (too_large) or in the surrogate range (surrogate).