Can't build GNU diffutils with emscripten

[vadimkantorov]

Getting really strange errors on Emscripten 2.0.5:

In file included from /mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.c:3:
/mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.h:78:60: error: expected expression
  return ((x << n) | (x >> ((CHAR_BIT * sizeof x) - n))) & SIZE_MAX;
                                                           ^
./stdint.h:558:21: note: expanded from macro 'SIZE_MAX'
#  define SIZE_MAX  _STDINT_MAX (0, , 0ul)
                    ^
./stdint.h:141:30: note: expanded from macro '_STDINT_MAX'
    (((((zero) + 1) << ((bits) ? (bits) - 1 - (signed) : 0)) - 1) * 2 + 1)
                             ^
In file included from /mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.c:3:
/mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.h:78:60: error: expected expression
./stdint.h:558:21: note: expanded from macro 'SIZE_MAX'
#  define SIZE_MAX  _STDINT_MAX (0, , 0ul)
                    ^
./stdint.h:141:39: note: expanded from macro '_STDINT_MAX'
    (((((zero) + 1) << ((bits) ? (bits) - 1 - (signed) : 0)) - 1) * 2 + 1)
In file included from /mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.c:3:
/mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.h:87:60: error: expected expression
./stdint.h:558:21: note: expanded from macro 'SIZE_MAX'
#  define SIZE_MAX  _STDINT_MAX (0, , 0ul)
                    ^
./stdint.h:141:39: note: expanded from macro '_STDINT_MAX'
    (((((zero) + 1) << ((bits) ? (bits) - 1 - (signed) : 0)) - 1) * 2 + 1)
                                      ^
                                      ^
4 errors generated.
emcc: error: '/mnt/c/Users/user/emsdk/upstream/bin/clang -target wasm32-unknown-emscripten -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=5 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -nostdsysteminc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/compat -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libc/musl/arch/emscripten -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/local/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SSE -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/neon -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/compiler-rt/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libunwind/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/cache/wasm/include -DEMSCRIPTEN -fignore-exceptions -fno-inline-functions -I. -I/mnt/c/Users/user/busyide/source/diffutils/lib -Oz -MT bitrotate.o -MD -MP -MF .deps/bitrotate.Tpo -c -o bitrotate.o /mnt/c/Users/user/busyide/source/diffutils/lib/bitrotate.c -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SDL -c -o bitrotate.o -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr' failed (1)
Makefile:1924: recipe for target 'bitrotate.o' failed
make[2]: *** [bitrotate.o] Error 1
make[2]: Leaving directory '/mnt/c/Users/user/busyide/build/wasm/diffutils/lib'
Makefile:1678: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/mnt/c/Users/user/busyide/build/wasm/diffutils/lib'
Makefile:1361: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
make: make

Full repro script:

set -e

ROOT=$PWD
CONFIGURE_wasm="emconfigure"
MAKE_wasm="emmake make"

URL_diffutils=https://ftp.gnu.org/gnu/diffutils/diffutils-3.7.tar.xz
CFLAGS_wasm_diffutils='-s ERROR_ON_UNDEFINED_SYMBOLS=0 -lidbfs.js -s WASM=1 -s SINGLE_FILE=1 -s MODULARIZE=1 -s EXPORT_NAME=busy -s FORCE_FILESYSTEM=1 -s EXPORTED_RUNTIME_METHODS=[\"FS\"] -s INVOKE_RUN=0 -Oz'
CACHE_wasm_diffutils=$ROOT/build/wasm-diffutils.cache

mkdir -p source/diffutils build/wasm/diffutils
wget --no-clobber $URL_diffutils -O source/$(basename $URL_diffutils)
tar -xf source/$(basename $URL_diffutils) --strip-components=1 --directory=source/diffutils

printf "ac_cv_search_nanosleep=yes\ngl_cv_func_nanosleep=no\ngl_cv_func_sleep_works=yes" > $ROOT/config.site

pushd build/wasm/diffutils
CONFIG_SITE=$ROOT/diffutils.site ${CONFIGURE_wasm} $ROOT/source/diffutils/configure --cache-file=${CACHE_wasm_diffutils} CFLAGS="${CFLAGS_wasm_diffutils}"
${MAKE_wasm}
popd

Log of configure step: configure.txt
Log of make step: make.txt

// generated ./build/wasm/diffutils/lib/stdint.h

/* size_t limit */
# undef SIZE_MAX
# if 0
#  ifdef _LP64
#   define SIZE_MAX  _STDINT_MAX (0, 64, 0ul)
#  else
#   define SIZE_MAX  _STDINT_MAX (0, 32, 0ul)
#  endif
# else
#  define SIZE_MAX  _STDINT_MAX (0, , 0ul)
# endif

// source ./source/diffutils/lib/stdint.in.h

/* size_t limit */
# undef SIZE_MAX
# if @APPLE_UNIVERSAL_BUILD@
#  ifdef _LP64
#   define SIZE_MAX  _STDINT_MAX (0, 64, 0ul)
#  else
#   define SIZE_MAX  _STDINT_MAX (0, 32, 0ul)
#  endif
# else
#  define SIZE_MAX  _STDINT_MAX (0, @BITSIZEOF_SIZE_T@, 0@SIZE_T_SUFFIX@)
# endif

So somehow @BITSIZEOF_SIZE_T@ is empty or undefined during configure?
Indeed, check for size of size_t failed at configure step (from configure.txt):

checking for bit size of ptrdiff_t...
/mnt/c/Users/user/busyide/source/diffutils/configure: 13901: test: =: unexpected operator
checking for bit size of size_t...
/mnt/c/Users/user/busyide/source/diffutils/configure: 13901: test: =: unexpected operator
checking for bit size of sig_atomic_t...
/mnt/c/Users/user/busyide/source/diffutils/configure: 13946: test: =: unexpected operator
checking for bit size of wchar_t...
/mnt/c/Users/user/busyide/source/diffutils/configure: 13946: test: =: unexpected operator
checking for bit size of wint_t...
/mnt/c/Users/user/busyide/source/diffutils/configure: 13946: test: =: unexpected operator

Here is the relevant snippet from source/diffutils/configure:

  for gltype in ptrdiff_t size_t ; do
    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for bit size of $gltype" >&5
printf %s "checking for bit size of $gltype... " >&6; }
if eval test \${gl_cv_bitsizeof_${gltype}+y}
then :
  printf %s "(cached) " >&6
else
  if ac_fn_c_compute_int "$LINENO" "sizeof ($gltype) * CHAR_BIT" "result"        "
  /* BSD/OS 4.0.1 has a bug: <stddef.h>, <stdio.h> and <time.h> must be
     included before <wchar.h>.  */
  #include <stddef.h>
  #include <signal.h>
  #if HAVE_WCHAR_H
  # include <stdio.h>
  # include <time.h>
  # include <wchar.h>
  #endif

#include <limits.h>"
then :

else
  result=unknown
fi

       eval gl_cv_bitsizeof_${gltype}=\$result

fi
eval ac_res=\$gl_cv_bitsizeof_${gltype}
           { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
printf "%s\n" "$ac_res" >&6; }
    eval result=\$gl_cv_bitsizeof_${gltype}
    if test $result = unknown; then
                                                result=0
    fi
    GLTYPE=`echo "$gltype" | tr 'abcdefghijklmnopqrstuvwxyz ' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ_'`
    printf "%s\n" "#define BITSIZEOF_${GLTYPE} $result" >>confdefs.h

    eval BITSIZEOF_${GLTYPE}=\$result
  done


  fi


  for gltype in sig_atomic_t wchar_t wint_t ; do
    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for bit size of $gltype" >&5
printf %s "checking for bit size of $gltype... " >&6; }
if eval test \${gl_cv_bitsizeof_${gltype}+y}
then :
  printf %s "(cached) " >&6
else
  if ac_fn_c_compute_int "$LINENO" "sizeof ($gltype) * CHAR_BIT" "result"        "
  /* BSD/OS 4.0.1 has a bug: <stddef.h>, <stdio.h> and <time.h> must be
     included before <wchar.h>.  */
  #include <stddef.h>
  #include <signal.h>
  #if HAVE_WCHAR_H
  # include <stdio.h>
  # include <time.h>
  # include <wchar.h>
  #endif

#include <limits.h>"
then :

else
  result=unknown
fi

       eval gl_cv_bitsizeof_${gltype}=\$result

fi

eval ac_res=\$gl_cv_bitsizeof_${gltype}
           { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
printf "%s\n" "$ac_res" >&6; }
    eval result=\$gl_cv_bitsizeof_${gltype}
    if test $result = unknown; then
                                                result=0
    fi
    GLTYPE=`echo "$gltype" | tr 'abcdefghijklmnopqrstuvwxyz ' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ_'`
    printf "%s\n" "#define BITSIZEOF_${GLTYPE} $result" >>confdefs.h

    eval BITSIZEOF_${GLTYPE}=\$result
  done


  for gltype in sig_atomic_t wchar_t wint_t ; do
    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking whether $gltype is signed" >&5
printf %s "checking whether $gltype is signed... " >&6; }
if eval test \${gl_cv_type_${gltype}_signed+y}
then :
  printf %s "(cached) " >&6
else
  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
/* end confdefs.h.  */

  /* BSD/OS 4.0.1 has a bug: <stddef.h>, <stdio.h> and <time.h> must be
     included before <wchar.h>.  */
  #include <stddef.h>
  #include <signal.h>
  #if HAVE_WCHAR_H
  # include <stdio.h>
  # include <time.h>
  # include <wchar.h>
  #endif

            int verify[2 * (($gltype) -1 < ($gltype) 0) - 1];
int
main (void)
{

  ;
  return 0;
}
_ACEOF
if ac_fn_c_try_compile "$LINENO"
then :
  result=yes
else

  result=no
fi

emcc -v
mcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 2.0.5
clang version 12.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.1485827954.workers.dev-llvm-llvm--project 5dbf80cad9556e222c4383960007fc0b27ea9541)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /mnt/c/Users/user/emsdk/upstream/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
shared:INFO: (Emscripten: Running sanity checks)

Building in-tree does not help either

[vadimkantorov]

These problems seem to repro on GitHub Actions. This is the config.site that I had to use to build it with Emscripten:

ac_cv_search_nanosleep=yes
gl_cv_func_nanosleep=no
gl_cv_func_sleep_works=yes
ac_cv_func_getprogname=no
ac_cv_func_wmempcpy=no
ac_cv_func_rawmemchr=no
ac_cv_func__set_invalid_parameter_handler=no
ac_cv_func_strcasecoll=no
ac_cv_func_stricoll=no
gl_cv_bitsizeof_ptrdiff_t=32
gl_cv_bitsizeof_sig_atomic_t=32
gl_cv_bitsizeof_size_t=32
gl_cv_bitsizeof_wchar_t=16
gl_cv_bitsizeof_wint_t=32

To sum up, Emscripten build can't detect absence of some functions (they're dettected, but are not linked successfully) and the sizeof of size_t types.

Hardcoding type sizes may shoot in the foot when wasm64 becomes a thing #12869

[sbc100]

If you'd like to improve our support for autoconf help would be much appreciated. We do some basic tests but clearly there are things we could improve.

One area I'd like to revisit is whether we should pretend to be a native compiler or not. I think today we pretend that we are native copiler with executable that can run the build host.. I think maybe we should be honest and admit to autoconf that we are a cross compiler.

[vadimkantorov]

I think these checks are executed fine during configure time. A question is why these functions link successfully during checks and do not link at actual compile-time?

[vadimkantorov]

Also, if I'm not mistaken, a few versions of Emscripten earlier just sleep hacks were needed.

[sbc100]

I'm not sure what you mean by "do not link at actual compile-time".. the errors above look like compile error no link errors.

I'm also not sure that you mean by "just sleep hacks were needed.". Are you saying things used to work and they don't? What sleep hacks are your referring too?

If there was a specific regression it would be great to know exactly which emscripten revision caused it.

[vadimkantorov]

1. I meant that autoconf checks seem to pass compilation successfully. And compilation fails at the "make" step. The fact that the check passed, but make step compilation did not is quite strange.

2. Yep, I think previously only the following were needed.

ac_cv_search_nanosleep=yes
gl_cv_func_nanosleep=no
gl_cv_func_sleep_works=yes

3. I'll try to find the old version where it worked with only sleep hacks, and will let you know

[sbc100]

BTW, hopefully you won't need the sleep at anymore as of a recent change : #12872

[vadimkantorov]

The sleep hacks were becasue of unsupported alarm() emulation...

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.