[SIMD] Support for Int8x16 and Int8x16List.

@mraleph

I'd like to start working on Int8x16 and Int8x16List so we can bring byte-level SIMD support to Dart.

The plan is:

Our Int32x4 implementation provides the blueprint for Int8x16 so I expect that implementing an Int8x16 type should mostly be a mechanical change that mirrors Int32x4.
Since our semantics make hoisting non-constant global values difficult (see: [vm] Failure to hoist certain static loads. #63390) I'd also like to try to make the Int8x16 constructor constant. Constant Symbol constructors provide the precedent. The change should be mechanical mirroring of Symbol.
Initial op set will be the minimum needed for indexOf. I have a prototype branch showing ~100x over Dart's current OneByteString.indexOf.

Goals:

Support Int8x16 and Int8x16List and dogfood it somewhere in the SDK. I think OneByteString.indexOf is probably the best use case that is simple enough and valuable enough to be both user-visible and benchmark-quantifiable.
Support accelerated byte-level SIMD operations on Wasm, x86 (gates on SSE availability) and arm64 & arm32 (gated on NEON availability).

Non goals:

js doesn't support SIMD operations. Using Wasm is not an option since crossing the js-wasm barrier introduces too much overhead, so Dart fallbacks will be used there and documentation will note that dart2wasm is the recommended target for SIMD performance.
riscv32 and riscv64 use fundamentally different SIMD instruction sets. To reduce the scope, Dart fallback implementations will be used on riscv targets. (some accelerated riscv ops may end up being added on demand if dogfooding requires them, but full parity with x86 and arm is explicitly not promised in this phase)

This will be the first step towards unblocking implementing more efficient json decoders (see: #35693) directly in Dart by eventually giving us the ability to use simdjson techniques.

I plan to continue adding support for more operations to Int8x16 driven by parsing-related use cases if this experiment goes well.

cc @mraleph @lrhn @kevmoo @rakudrama thoughts?

(1) Thank you SO MUCH for your interest here!
(2) Thank you SO MUCH for making a clear proposal FIRST before sending complex CLs (I hate that we have a different name for them than PRs, whatever)
(3) I have no "product" objections. Glad you're thinking about Wasm here. I think your approach w/ JS is fine. Forever our typed data has mostly been modeled off what's possible in https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray – I think it's good to think beyond it.

I'll leave it up to the engineers that you CC'd to give a go-ahead to proceed. It's ideal that we come to some agreement about design/scope before you start sending CLs, although if you get things working on your own machine and you have initial thoughts, performance numbers, that's great to see!

Main objection would be that they are not directly supported by JS.
But then, neither is Int64List. (Is Int32x4?)

Using it for OneByteString requires a way to get the bytes of a string as a ByteBuffer. That could leak the implementation of it treats one- and two-byte strings differently, so possibly not something that can be publicly available.
(And if it's only internal, it doesn't need to be defined in Dart.)

I have no real problem with the idea if the implementeres can agree.

And if it's only internal [...]

Yes, exactly. It would be used within the SDK and not implemented in user space.

Is Int32x4?

Yes, with dart2js it compiles to a scalar variant.

While I am not per-se opposed to adding Int8x16 and associated types, I am concerned that approach we have chosen with existing SIMD types does not scale:

It requires us to add types to core libraries (what if we want to support 256 wide SIMD types? what if we want to support 512 wide SIMD types?)
It provides inconsistent portable API for something that is fundamentally non-portable: because SIMD instruction sets can be very different between specific architectures.
There is no way to feature detect underlying CPU capabilities and write non-portable code which properly utilizes underlying hardware/software capabilities.

If we want to land something for core libraries then I think we should start by keeping necessary additions private (as sort of hidden types and helpers) rather than adding them as public types.

In the long run I think we should seriously consider deprecating existing SIMD types and developing new SIMD APIs which are target specific instead.

Initial op set will be the minimum needed for indexOf. I have a prototype branch showing ~100x over Dart's current OneByteString.indexOf.

I don't know what benchmark is - but I would imagine that just calling existing highly optimized C function is possibly a better strategy then porting all of this to Dart. The same with JSON parsing.

@mraleph

@mraleph All fair points, here are my thoughts:

Width concerns

It requires us to add types to core libraries (what if we want to support 256 wide SIMD types? what if we want to support 512 wide SIMD types?)

There are two categories we need to differentiate when talking about SIMD: fixed length and variable length.

fixed length: e.g., NEON on arm / SSE on x86 / wasm
variable length: e.g., SVE on arm / RVV on riscv

Fixed length is well supported by most of our users (arm & wasm)
Variable length is not, e.g., SVE is not practically available in the app-developer toolchains on Apple or recent Android targets, even where the hardware supports it.

When it comes to fixed length SIMD support, I think it's unlikely that bigger lanes are going to be relevant to us in the near future.

In practice, only riscv and x86 support lanes wider than 128-bit
NEON was introduced in 2007 so it seems unlikely that we are going to get bigger lanes in the near future on arm
Wasm only supports 128-bit
Most of our users are already on or moving to arm and wasm (given that x86 is losing relevance and Flutter is moving towards wasm)

Focusing on what's supported everywhere (128-bit) is suboptimal for server x86 use-cases, but covers essentially all Dart users.

Future direction

I discussed this briefly with @kevmoo (who has started exploring improvements to our SIMD story in https://dart-review.googlesource.com/c/sdk/+/481980 and https://github.com/kevmoo/sdk/blob/simd_proposal/SIMD_proposal.md). If we want to have an optimal fixed-length SIMD story, I think a layered approach would be good and here's what it could look like (this is not a proposal, but a future direction we could take that is compatible with our current approach):

Architecture-level base types (SimdNeon, SimdSse, SimdRvv, SimdWasm)
Width-level base types (that depend on architecture-level base types), e.g., Simd128, Simd256, Simd512.
Lane-level base types (that depend on width-level base types), e.g., Int8x?, Int16x?, Int32x?.
High-level procedures (that depend on lane-level base types), e.g., nibbleSearch, byteSearch, simdBinarySearch.

Dart lacks type-level values, so we can't easily abstract over Int8x?, so I think it's reasonable to work with what we have right now, and expand it to Int8x32/Int8x64 in the future if needed.

Compatibility concerns

It provides inconsistent portable API for something that is fundamentally non-portable: because SIMD instruction sets can be very different between specific architectures.

Agreed, but in practice arm and wasm users get full benefit from a 128-bit API. Here are numbers from my POC for OneByteString.indexOf:

today we get 2GB/s
memchr is 60GB/s
A vectorized 4x unrolled NEON implementation in C manages 120GB/s
we can reach 90GB/s with byte-level SIMD support in Dart

This is a HUGE improvement and these wins will be able to cascade into other parts of the SDK and ecosystem: LineSplitter, hex/base64/json parsing.

but I would imagine that just calling existing highly optimized C function is possibly a better strategy then porting all of this to Dart. The same with JSON parsing.

Yes, but that optimizes String.indexOf specifically. The proposal here is to unlock byte-level SIMD as a primitive, with String.indexOf as one of many beneficiaries.

Most of our users are already on or moving to arm and wasm (given that x86 is losing relevance and Flutter is moving towards wasm)

The number of times I've heard "The PC is dead" makes me question that assumption.

If nothing else, people will be using x64-based PCs while developing.

[vm/compiler] Intrinsify Int32x4 integer lane getters (x/y/z/w).

[dart2wasm] Use WasmV128 for I32x4 backing storage

The proposal here is to unlock byte-level SIMD as a primitive, with String.indexOf as one of many beneficiaries.

I'm not clear on how Int8x16 helps with String.indexOf. There is no API to view the string bytes as typed data, and no guarantees as to the alignment of the data.

@rakudrama

There is no API to view the string bytes as typed data

@rakudrama That's right, users can't and shouldn't, but we have more options from within the sdk. (Note that this is meant to just be a simple motivating example that everyone can directly benefit from, but there are plenty more that users will be able to write themselves.)

@mraleph

There is no API to view the string bytes as typed data

@rakudrama That's right, users can't and shouldn't, but we have more options from within the sdk. (Note that this is meant to just be a simple motivating example that everyone can directly benefit from, but there are plenty more that users will be able to write themselves.)

Is there a simple motivating example that is actually feasible? Providing Int8x16 is not going to make String.indexOf faster.

AFAIK there is no requirement for Uint8List to have (or be used as) a buffer with 16-byte alignment.

Generally, it is safe only to create views with same-size-or-smaller elements. It doesn't matter much for scalar values, since there is some minimum word-alignment, but it would be wasteful to demand that an in-heap new Uint8List(32) has 16-byte or 32-byte alignment at the first byte of the data just in case it might be used as a buffer for some Int8x16List or some future 256-bit SIMD view.

The alignment considerations add weight to @mraleph's concern that our approach to SIMD is not scalable and should be reconsidered.

[dart2wasm] Unify unmodifiable typed data mixins

@mraleph

I share similar concerns to @mraleph 's #63471 (comment) .

On a high level I would want us to have two layers:

Expose the full power of HW/backend-platform to Dart users: We have been working towards this goal on multiple fronts:
- On the VM: shared memory, dart:ffi, allow packages to include native code via build hooks.
- On wasm: We have a dart:_wasm library that we eventually intend to extend and make public / usable by end users (issue)
  => These things are somewhat outside the Dart language, where a subset of Dart is a DSL to express C code and Wasm code with compile-time errors if one uses them wrong.
  => Users could then make packages that depend on these low level, backend-specific libraries and possibly use conditional imports to abstract over them across backends.
Make the core dart:* libraries (i.e. excluding backend specific ones like dart:ffi) be nice to use, feature rich and performant.
=> These are Dart language features.
=> We try not to put things into those libraries that a) only work on some backends but not all or b) only perform well on backends and terribly on others

These two differ materially: For example dart:typed_data abstractions are never "optimal": For example

they always perform bounds checks (which may not always be eliminatable by compilers)
they may have extra wrapper objects (e.g. Int32xList is a wrapper object around a WasmArray object -- 2 objects in the heap)
they may be polymorphic (e.g. Uint8List could be backed by different implementations and as a result a call site cannot always devirtualize [] / []= calls and make them simple byte loads/stores -- e.g. in wasm)

=> So if one wants to get the most performance / size - the pure power of HW - then high level dart:typed_data API may actually not be the best to get there.

We have some other requests similar to this Int8x16 that somewhat fall in the middle between these two - where there is tension whether it should be part of the language / core libraries or some backend specific functionality:

Float16List, BFloat16List ([typed_data] Add Float16List + [vm/ffi] Add Float16 #56319)
int16, int32, ... (proposal to introduce a rich, efficient and more flexible type system in dart 3.x #51994)

My view is that we can put something in to common core libraries iff

we have clear evidence it's useful / needed and cannot be done in a package
we have clear evidence it makes things better (e.g. more performant)
it works well across all backends

[dart2wasm] Back Int32x4List with WasmArray<WasmV128>

[typed_data] Add Int32x4.equal and anyTrue

@rakudrama

@rakudrama Is there a simple motivating example that is actually feasible?

Here is one from user space, if it helps. I'm porting libwebp's VP8 encoder to Dart. Its inner kernels are sums of squared differences, a Hadamard distortion metric, a 4x4 DCT and a quantizer, all reading 8-bit samples out of a macroblock cache. libwebp has NEON and SSE versions of all of them.

Unlike String.indexOf, the data is already a Uint8List that the codec allocates and lays out itself, so there is no missing view API and nothing internal to expose. The layout is the one libwebp uses: rows 32 bytes apart, 16 luma rows starting at offset 0. Every 16-byte row is therefore already 16-byte aligned and an Int8x16List.view over it would work as is.

The alignment concern does apply to half of it. The 4x4 sub-blocks sit at offsets 0, 4, 8 and 12 within a row, so three out of four are not 16-aligned. Those kernels would need a 16-byte load from an arbitrary offset, or a way to fill lanes from four separate 4-byte groups.

What the byte path costs today, with Float32x4, which is the only thing available now. dart compile exe, macos_arm64:

	scalar	Float32x4
`sse16x16`	1.730 us	1.480 us
`hadamard4x4`	0.191 us	0.245 us

The Hadamard kernel loses even though everything it needs is available and in range. The transform is separable and its passes commute, so it needs a single 4x4 transpose, done with the usual eight shuffleMix. What is left is filling four vectors from sixteen strided bytes, and on a 4x4 block that costs more than four-wide arithmetic saves. For reference, the same arithmetic over an already populated Float32x4List runs 2.5x faster than the scalar loop, so that is roughly the ceiling a cheap byte to lane path would be buying.

So for this workload the value of a byte lane type is not the byte arithmetic. It is that sixteen byte loads and four vector constructions collapse into one load.

Both kernels, in the shape of benchmarks/SimdInt32x4. main checks the two variants agree before either is timed.

import 'dart:typed_data';

import 'package:benchmark_harness/benchmark_harness.dart';

/// Row stride of the block cache, as in libwebp.
const int bps = 32;

abstract class KernelBench extends BenchmarkBase {
  KernelBench(String name) : super('SimdImageKernels.$name');

  late final Uint8List a;
  late final Uint8List b;

  @override
  void setup() {
    a = Uint8List(bps * 32);
    b = Uint8List(bps * 32);
    var seed = 12345;
    for (int i = 0; i < a.length; i++) {
      seed = (seed * 1103515245 + 12345) & 0x7fffffff;
      a[i] = (seed >> 16) & 0xff;
      seed = (seed * 1103515245 + 12345) & 0x7fffffff;
      b[i] = (seed >> 16) & 0xff;
    }
  }
}

int sse16x16Scalar(Uint8List a, Uint8List b) {
  int count = 0;
  int aOff = 0, bOff = 0;
  for (int y = 0; y < 16; y++) {
    for (int x = 0; x < 16; x++) {
      final int d = a[aOff + x] - b[bOff + x];
      count += d * d;
    }
    aOff += bps;
    bOff += bps;
  }
  return count;
}

double sse16x16Simd(Uint8List a, Uint8List b) {
  Float32x4 acc = Float32x4.zero();
  int aOff = 0, bOff = 0;
  for (int y = 0; y < 16; y++) {
    for (int x = 0; x < 16; x += 4) {
      final Float32x4 d = Float32x4(
        0.0 + (a[aOff + x] - b[bOff + x]),
        0.0 + (a[aOff + x + 1] - b[bOff + x + 1]),
        0.0 + (a[aOff + x + 2] - b[bOff + x + 2]),
        0.0 + (a[aOff + x + 3] - b[bOff + x + 3]),
      );
      acc += d * d;
    }
    aOff += bps;
    bOff += bps;
  }
  return acc.x + acc.y + acc.z + acc.w;
}

class Sse16x16Scalar extends KernelBench {
  Sse16x16Scalar() : super('sse16x16Scalar');
  int sink = 0;
  @override
  void run() => sink += sse16x16Scalar(a, b);
}

class Sse16x16Simd extends KernelBench {
  Sse16x16Simd() : super('sse16x16Simd');
  double sink = 0;
  @override
  void run() => sink += sse16x16Simd(a, b);
}

const List<int> weights = <int>[
  38, 32, 20, 9,
  32, 28, 17, 7,
  20, 17, 10, 4,
  9, 7, 4, 2,
];

int hadamard4x4Scalar(Uint8List src) {
  final Int32List t = Int32List(16);
  int off = 0;
  for (int y = 0; y < 4; y++) {
    final int a0 = src[off] + src[off + 2];
    final int a1 = src[off + 1] + src[off + 3];
    final int a2 = src[off + 1] - src[off + 3];
    final int a3 = src[off] - src[off + 2];
    t[y * 4] = a0 + a1;
    t[y * 4 + 1] = a3 + a2;
    t[y * 4 + 2] = a3 - a2;
    t[y * 4 + 3] = a0 - a1;
    off += bps;
  }
  int sum = 0;
  for (int i = 0; i < 4; i++) {
    final int a0 = t[i] + t[8 + i];
    final int a1 = t[4 + i] + t[12 + i];
    final int a2 = t[4 + i] - t[12 + i];
    final int a3 = t[i] - t[8 + i];
    final int b0 = a0 + a1;
    final int b1 = a3 + a2;
    final int b2 = a3 - a2;
    final int b3 = a0 - a1;
    sum += weights[i] * (b0 < 0 ? -b0 : b0) +
        weights[i + 4] * (b1 < 0 ? -b1 : b1) +
        weights[i + 8] * (b2 < 0 ? -b2 : b2) +
        weights[i + 12] * (b3 < 0 ? -b3 : b3);
  }
  return sum;
}

final Float32x4 w0 = Float32x4(38, 32, 20, 9);
final Float32x4 w1 = Float32x4(32, 28, 17, 7);
final Float32x4 w2 = Float32x4(20, 17, 10, 4);
final Float32x4 w3 = Float32x4(9, 7, 4, 2);

double hadamard4x4Simd(Uint8List src) {
  Float32x4 row(int off) => Float32x4(
        0.0 + src[off],
        0.0 + src[off + 1],
        0.0 + src[off + 2],
        0.0 + src[off + 3],
      );
  final Float32x4 r0 = row(0);
  final Float32x4 r1 = row(bps);
  final Float32x4 r2 = row(2 * bps);
  final Float32x4 r3 = row(3 * bps);

  Float32x4 a0 = r0 + r2;
  Float32x4 a1 = r1 + r3;
  Float32x4 a2 = r1 - r3;
  Float32x4 a3 = r0 - r2;
  final Float32x4 g0 = a0 + a1;
  final Float32x4 g1 = a3 + a2;
  final Float32x4 g2 = a3 - a2;
  final Float32x4 g3 = a0 - a1;

  final Float32x4 t0 = g0.shuffleMix(g1, 0x44);
  final Float32x4 t2 = g0.shuffleMix(g1, 0xEE);
  final Float32x4 t1 = g2.shuffleMix(g3, 0x44);
  final Float32x4 t3 = g2.shuffleMix(g3, 0xEE);
  final Float32x4 h0 = t0.shuffleMix(t1, 0x88);
  final Float32x4 h1 = t0.shuffleMix(t1, 0xDD);
  final Float32x4 h2 = t2.shuffleMix(t3, 0x88);
  final Float32x4 h3 = t2.shuffleMix(t3, 0xDD);

  a0 = h0 + h2;
  a1 = h1 + h3;
  a2 = h1 - h3;
  a3 = h0 - h2;
  final Float32x4 s = (a0 + a1).abs() * w0 +
      (a3 + a2).abs() * w1 +
      (a3 - a2).abs() * w2 +
      (a0 - a1).abs() * w3;
  return s.x + s.y + s.z + s.w;
}

class Hadamard4x4Scalar extends KernelBench {
  Hadamard4x4Scalar() : super('hadamard4x4Scalar');
  int sink = 0;
  @override
  void run() => sink += hadamard4x4Scalar(a);
}

class Hadamard4x4Simd extends KernelBench {
  Hadamard4x4Simd() : super('hadamard4x4Simd');
  double sink = 0;
  @override
  void run() => sink += hadamard4x4Simd(a);
}

void main() {
  final Uint8List a = Uint8List(bps * 32);
  final Uint8List b = Uint8List(bps * 32);
  int seed = 12345;
  for (int i = 0; i < a.length; i++) {
    seed = (seed * 1103515245 + 12345) & 0x7fffffff;
    a[i] = (seed >> 16) & 0xff;
    seed = (seed * 1103515245 + 12345) & 0x7fffffff;
    b[i] = (seed >> 16) & 0xff;
  }
  if (sse16x16Scalar(a, b) != sse16x16Simd(a, b).toInt() ||
      hadamard4x4Scalar(a) != hadamard4x4Simd(a).toInt()) {
    throw StateError('scalar and SIMD variants disagree');
  }

  Sse16x16Scalar().report();
  Sse16x16Simd().report();
  Hadamard4x4Scalar().report();
  Hadamard4x4Simd().report();
}

Navigation Menu

Repository navigation

[SIMD] Support for Int8x16 and Int8x16List. #63471

Goals:

Non goals:

Width concerns

Future direction

Compatibility concerns

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[SIMD] Support for Int8x16 and Int8x16List. #63471

Description

Goals:

Non goals:

Activity

kevmoo commented on May 28, 2026

lrhn commented on May 28, 2026

modulovalue commented on May 28, 2026

mraleph commented on May 29, 2026

mraleph commented on May 29, 2026

modulovalue commented on May 29, 2026

Width concerns

Future direction

Compatibility concerns

lrhn commented on May 30, 2026

rakudrama commented on Jul 15, 2026

modulovalue commented on Jul 15, 2026

rakudrama commented on Jul 24, 2026

mkustermann commented on Jul 29, 2026

krll-kov commented on Sep 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions