Bad double -> uint32 conversion when using clamp trap mode

[kumpera]

Test case:

#include <stdio.h>
#include <stdlib.h>

static double *bla;

int main (int argc, char *argv[]) {
	int i = 0;
	bla = malloc (8 * 16);
	for (i = 0; i < 16; ++i)
		bla[i] = 4294967295.0;
	unsigned int x = (unsigned int)bla[6];
	printf ("x is %f x is %x\n", bla[6], x);
	return 0;
}

How to compile it:

emcc -s "BINARYEN_TRAP_MODE='clamp'" -Os -g  -s WASM=1 repro.c -o repro.html

It prints x is 4294967295.000000 x is 80000000

The expected value for x is ffffffff and it works as expected if BINARYEN_TRAP_MODE is not passed.

[kripken]

Thanks, yeah, looks like in clamp mode we currently just clamp to [-2147483648..2147483648), which is indeed odd for an unsigned conversion. We should clamp to unsigned values when the operation is an unsigned one.

@jgravelle-google, this would change the clamping code a little, so it could conflict with WebAssembly/binaryen#1168, should I wait for that to land first?

[jgravelle-google]

I did notice that while working through that. I can add it to that PR if you don't mind

[kripken]

Sounds good. Meanwhile I did a little experimenting and wrote this patch, so that might help (or feel free to ignore and do it your way).