Skip to content

Support an efficient general purpose hashing function via XXH3-64. #64359

Open

Description

@modulovalue

It's difficult to implement an efficient XXH3 in Dart, which is an efficient, non-cryptographic, general purpose hashing function.

(For the purposes of this issue, efficient is defined as reaching 80% of the performance of a native implementation on a recent arm64 machine.)

This has come up in #64065 where several people have expressed a more efficient hashing function could be useful for native hooks (cc @dcharkes) & package:analyzer (cc @scheglov).

Here are some rough numbers on an arm64 M5 machine:

With some optimizations from @mraleph and others we were able to achieve up to 5 GB/s. By eliminating some indirection we will be able to get up to 9 GB/s.

A vectorized implementation manages 50 GB/s. I have some patches (called downstream in the chart below) that push us towards 13 GB/s-ish.

(For comparison, reasonable upper bounds for other hashes are: md5 -> 1 GB/s, BLAKE3 -> 3 GB/s, CRC32 -> 40 GB/s)

Image

Consider an efficient implementation like this one and notice that the main loop is vectorized. Good news, it looks like all of those are part of the Wasm SIMD API surface and therefore part of #64170.

Given the performance improvements of #64337, this looks like a useful stress test that even exercises some more obscure operations and I think the target of 80% is reasonable. The open question is how close appropriate SIMD primitives are going to get us.

Further investigations are blocked on an Int64x2 API.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions