Rounding operators

[fdwr]

Summary

While writing out the emulation decomposition for quantizeLinear, I lacked rounding functions in WebNN, particularly the default IEEE standard round-tied-halves-to-nearest-even. There's not a nice way to simulate this missing function from existing WebNN operators, and it's a basic primitive that belongs in a library anyway. So I propose filling in some gaps ⭐:

• ➕ Add roundEven() which is the IEEE standard's default, and relevant ML libraries support it (see support below).
• ➕ Document trunc via emulation at least. I originally thought it was worth adding, but I haven't actually seen it in models, and evidently only DML ROUND and PyTorch torch.trunc have it, not TFLite/CoreML/ONNX. There are emulations below.

Known Rounding Modes

Rounding mode	WebNN	IEEE	C++ equivalent	JS equivalent
RHE nearest int, halves to even ⭐ (banker's rounding)	❌	convertToIntegerTiesToEven(x) The recommended IEEE float default	C23 roundeven std::nearbyint FE_TONEAREST	❌😲??
RHAZ nearest int, halves away from zero	❌ floor(abs(x) + 0.5) * sign(x)	convertToIntegerTiesToAway(x)	std::round	❌
RHTZ nearest int, halves toward zero	❌ ceil(abs(x) - 0.5) * sign(x)	❌	❌	❌
RHU nearest int, halves up	❌ floor(x + 0.5)	❌	❌	Math.round
RHD nearest int, halves down	❌ ceil(x - 0.5)	❌	❌	❌
RAZ away from zero	❌ ceil(abs(x) * sign(x))	❌	❌	❌
RTZ toward zero (trunc) ⭐	❌ floor(abs(x) * sign(x))	convertToIntegerTowardZero(x)	std::trunc std::nearbyint FE_TOWARDZERO	Math.trunc
RU up to positive infinity (ceil)	✅ ceil(x)	convertToIntegerTowardPositive(x)	std::ceil std::nearbyint FE_UPWARD	Math.ceil
RD down to negative infinity (floor)	✅ floor(x)	convertToIntegerTowardNegative(x)	std::floor std::nearbyint FE_DOWNWARD	Math.floor

Note

• Some of these modes are not generally useful, but they are very useful within their domain. For example, RHD is used often in graphics when rasterizing triangles or resampling images rather than RHE or other modes to avoid staggered/asymmetric pattern artifacts (WebNN's resample uses RHD internally). I'm not proposing those though.
• Javascript's Math.round is oddly aberrant, being neither RHE or RHAZ (the two most common IEEE nearest modes). So beware when using it as a baseline reference. Also I'm using "up" (higher value toward positive infinity) and "down" (lower value toward negative infinity) like C's usage, not Java's bidirectional RoundingMode where up and down mean away from zero and toward zero.

Library support

• RHE nearest int, halves to even:
  • DML ROUND with DML_ROUNDING_MODE_HALVES_TO_NEAREST_EVEN
  • coremltools.converters.mil.mil.ops.defs.iOS15.elementwise_unary.round 🤔 *It's not clearly RHE from the docs, but its single number example of 0.5 implies that it is, and by default, I presume the IEEE default. Otherwise it should be emulatable via elementwise_binary.sub(x, elementwise_binary.mod(x, 1.0f)). 🤞
  • tfl.round (TFL::RoundOp) 🤔 *It's not clearly documented for TFLite, but I'm presuming TFL follows TF in using RHE.
  • ONNX Round
  • PyTorch torch.round
• RTZ toward zero (trunc):
  • DML ROUND with DML_ROUNDING_MODE_TOWARD_ZERO
  • PyTorch torch.trunc prims.trunc

Examples

Original value	-2.5	-1.75	-1.5	-1.25	0	1.25	1.5	1.75	2.5
RHE nearest int, halves to even	-2	-2	-2	-1	0	1	2	2	2
RHAZ nearest int, halves away from zero	-3	-2	-2	-1	0	1	2	2	3
RHTZ nearest int, halves toward zero	-2	-2	-1	-1	0	1	1	2	2
RHU nearest int, halves up	-2	-2	-1	-1	0	1	2	2	3
RHD nearest int, halves down	-3	-2	-2	-1	0	1	1	1	2
RAZ away from zero	-3	-2	-2	-2	x	2	2	2	3
RTZ toward zero (trunc)	-2	-1	-1	-1	0	1	1	1	2
RU up to positive infinity (ceil)	-2	-1	-1	-1	0	2	2	2	3
RD down to negative infinity (floor)	-3	-2	-2	-2	0	1	1	1	2

Emulation

RHE (roundEven)

A backend can emulate RHE if it has modulus/remainder and trunc, as x - std::remainder(x, static_cast<T>(1)) (source) (even though WebNN itself lacks both those operators).

Another odd approach that evidently works for float32 (but not as-is for float16/float64) uses a few condition checks with addition and subtraction, but that ends up being a clumsy construction in WebNN.

float roundHalvesToNearestEven(float f)
{
    if (!(f > -8388608.0f && f < 8388608.0f)) // Return true for NaN
        return f;
    else if (f > 0)
        return float(f + 8388608.0f) - 8388608.0f;
    else
        return float(f - 8388608.0f) + 8388608.0f;
}

// output = abs(input) < magicValue ? (abs(input) + magicValue - magicValue) * sign(input) : input
function roundEven(builder, input)
{
    const magicValue = builder.const("float32", 8388608.0f);
    const absInput = builder.abs(input);
    builder.where(
        builder.less(absInput, magicValue),
        builder.mul(
            build.sub(builder.add(absInput, magicValue), magicValue),
            builder.sign(input)
        ),
        input
    );
}

RTZ (trunc)

// output = floor(abs(input)) * sign(input)
function trunc(builder, input)
{
    return builder.mul(
        builder.floor(builder.abs(input)),
        builder.sign(input)
    );
}

There's also an approximation using a round-trip cast (float -> int -> float), but that's lossy with large numbers.

Models

• RHE:
  • tiny-yolov3-11\yolov3-tiny.onnx
  • QWNet
• RTZ
  • ? (none personally known)

API

partial interface MLGraphBuilder {
  MLOperand roundEven(MLOperand input, optional MLOperatorOptions options = {});
};

partial dictionary MLOpSupportLimits {
  MLSingleInputSupportLimits roundEven;
};

operand	allowed data types	allowed ranks
input	"float32", "float16"	N
output	same as input	same as input

Naming: Why not just "round"? round is woefully ambiguous as evidenced from the differences between C++ std::round (RHAZ) vs Javascript/Java Math.round (RHU) vs round in others like HLSL (RHE) 😕. roundEven (like C23 roundeven) is directly clear in the name.

[fdwr]

@mwyrzykowski Can you clarify with one of your contacts if CoreML's elementwise rounding is round-to-nearest-even? I wager so, but can't quite tell from the wording "Return the round value of the input x to nearest integer, element-wise. 0.5 is rounded to 0.". TY.

[mwyrzykowski]

@mwyrzykowski Can you clarify with one of your contacts if CoreML's elementwise rounding is round-to-nearest-even? I wager so, but can't quite tell from the wording "Return the round value of the input x to nearest integer, element-wise. 0.5 is rounded to 0.". TY.

@fdwr I am still discussing but it appears the documentation might be inaccurate. Essentially, it seems implementation defined and round(X.5) may result in either ceil(X.5) or floor(X.5) depending on which backend (MPS, BNNS, ANE, CPU, etc) the model ends up running on.

This inconsistency could likely be confirmed via testing, I haven't gotten around to it.

Safer thing in the specification initially would be to leave round(X.5) as implementation defined.

[fdwr]

This inconsistency could likely be confirmed via testing

Thanks Mike. Since mil.ops.defs.iOS17.quantization_ops.quantize utilizes rounding in its definition (quantized_data = clip(round(input / scale) + zero_point)), when testing, could you also try quantize with input = [1.5, 2.5], scale = 1.0, zeroPoint = 0, axis = 0? It would be informative.

Safer thing in the specification initially would be to leave round(X.5) as implementation defined

🤔 Hmm, while some aspects I'm comfortable with leaving as implementation-defined (like how gather treats invalid indices, the order of summation for convolution filters, the exact precision of complex operators...), something this primitive I'd really prefer to be precisely defined, even if it has to be via emulation (perhaps some combination of elementwise_binary.mod and elementwise_unary.cast or elementwise_binary.floor_div...), but further testing will reveal our constraints better 🤞...

[huningxin]

I happened to test round on coreML, for input = [1.5, 2.5], the output is [2., 2.].

[fdwr]

I happened to test round on coreML, for input = [1.5, 2.5], the output is [2., 2.].

Thanks, so round-halves-to-nearest-even behavior👍. Could you (or Mike or Phillis) try -1.5 and -2.5 too, if you still have the machine and test case available? Also from @mwyrzykowski per the W3C meeting, we should try different MLComputeUnits.

[huningxin]

Could you (or Mike or Phillis) try -1.5 and -2.5 too

Yes, the results are -2., -2.

we should try different MLComputeUnits.

I tried to set different compute unit (coremltools.ComputeUnit.CPU_ONLY, coremltools.ComputeUnit.CPU_AND_GPU and coremltools.ComputeUnit.CPU_AND_NE), all give the same results [-2., -2., 2., 2.] for inputs [-1.5, -2.5, 1.5, 2.5].

@inexorabletash mentioned a small program may always run on CPU. I am not sure whether this is the case. /cc @philloooo

My test code for reference:

from coremltools.converters.mil import Builder as mb
import coremltools as ct
import numpy as np

@mb.program(input_specs=[mb.TensorSpec(shape=(4,)),])
def prog(x):
    x = mb.round(x=x, name='round')
    return x

model = ct.convert(prog, compute_units=ct.ComputeUnit.CPU_AND_NE)

prediction = model.predict({
  'x': np.array([-1.5, -2.5, 1.5, 2.5], dtype=np.float32),
})

print(prediction)

[mwyrzykowski]

@inexorabletash mentioned a small program may always run on CPU. I am not sure whether this is the case. /cc @philloooo

MLComputePlan will indicate which unit the model actually ran on - https://developer.apple.com/documentation/coreml/mlcomputeplan-85vdw

       // Get the compute device usage for the operation.
       MLComputeDeviceUsage *computeDeviceUsage = [computePlan computeDeviceUsageForMLProgramOperation:operation];
       // Get the estimated cost of executing the operation.
       MLComputePlanCost *estimatedCost = [computePlan estimatedCostOfMLProgramOperation:operation];

[huningxin]

After using a large size tensor (in shape [1, 1000, 1000, 4]) and stacking 5 round operators into the model, I managed to run the round operators on NPU. I can confirm that by inspecting the MLComputeDeviceUsage.preferred_compute_device of each round operator. The rounding behavior is inconsistent regarding to CPU. E.g. for input [-1.5, -2.5, 1.5, 2.5], the output is [-2. -3. 2. 3.] on NPU that seems to use RAZ (away from zero).

@fdwr

but can't quite tell from the wording "Return the round value of the input x to nearest integer, element-wise. 0.5 is rounded to 0.".

As @mwyrzykowski mentioned, the doc may be inaccurate. It is true when the compute unit is CPU. If it is NPU, 0.5 is rounded to 1.

[fdwr]

NPU that seems to use RAZ (away from zero).

Only halves, right? (RAZ vs RHAZ) So [-1.1, -1.9, 1.1, 1.9] would presumably be [-1, -2, 1, 2].

If it is NPU, 0.5 is rounded to 1.

Doh, I wish it had rounded using the IEEE default which reduces accumulated bias - thanks for researching. For something as low-level as rounding, I'd want it to have well-defined behavior, which implies emulation; and since it seems we can't know/dictate ahead of time which device the rounding operator will execute on, then maybe we would always have to decompose roundEven on the CoreML backend, even if it would end up going through the GPU 🫤? If though we knew that node would end up going through the NPU, we might be able to utilize a shorter emulation by using the NPU's roundRhaz like the round(x) = roundRhaz(x - sign(x) / 2), whereas no emulation on the GPU code path, or maybe round(x) = elementwise_binary.sub(x, elementwise_binary.mod(x, 1.0f)) would work nicely for both (only 2 ops).

[fdwr]

I'll create the roundEven PR ⏳ (per discussion at https://github.com/webmachinelearning/meetings/blob/main/telcons/2025-05-22-wg-agenda.md).