Included zlib is built with optional instructions on aarch64

[JulianGro]

Version

18.16.0 and 18.15.0

Platform

Linux c9869ce0a7f0 5.15.76-v8+ #1597 SMP PREEMPT Fri Nov 4 12:16:41 GMT 2022 aarch64 aarch64 aarch64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

No response

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

It should check the host CPU for the available instructions and/or there should be a command line switch for building without aes instructions.

What do you see instead?

The included Zlib is hardcoded to be built with -march=armv8-a+aes+crc on AArch64. Common AArch64 devices such as the Raspberry Pi 4 don't support aes instructions.

Additional information

Changing this in line 69 (v18.16.0) of deps/zlib/zlib.gyp to

                  'cflags': [ '-march=armv8-a+crc' ],

throws Assembler errors:

  cc -o /node/out/Release/obj.target/zlib_adler32_simd/deps/zlib/adler32_simd.o ../deps/zlib/adler32_simd.c '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_THREADS' '-DADLER32_SIMD_NEON' -I../deps/zlib  -msign-return-address=all -pthread -Wall -Wextra -Wno-unused-parameter -fPIC -O3 -fno-omit-frame-pointer  -MMD -MF /node/out/Release/.deps//node/out/Release/obj.target/zlib_adler32_simd/deps/zlib/adler32_simd.o.d.raw   -c
cc: warning: switch '-msign-return-address=all' is no longer supported
  cc -o /node/out/Release/obj.target/zlib_arm_crc32/deps/zlib/crc32_simd.o ../deps/zlib/crc32_simd.c '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_THREADS' '-DCRC32_ARMV8_CRC32' '-DARMV8_OS_LINUX' -I../deps/zlib  -msign-return-address=all -pthread -Wall -Wextra -Wno-unused-parameter -fPIC -march=armv8-a+crc -O3 -fno-omit-frame-pointer  -MMD -MF /node/out/Release/.deps//node/out/Release/obj.target/zlib_arm_crc32/deps/zlib/crc32_simd.o.d.raw   -c
cc: warning: switch '-msign-return-address=all' is no longer supported
  g++ -o /node/out/Release/obj.target/gtest/deps/googletest/src/gtest-assertion-result.o ../deps/googletest/src/gtest-assertion-result.cc '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_THREADS' '-DGTEST_HAS_POSIX_RE=0' '-DGTEST_LANG_CXX11=1' -I../deps/googletest -I../deps/googletest/include  -msign-return-address=all -pthread -Wall -Wextra -Wno-unused-parameter -fPIC -O3 -fno-omit-frame-pointer -fno-rtti -fno-exceptions -std=gnu++17 -MMD -MF /node/out/Release/.deps//node/out/Release/obj.target/gtest/deps/googletest/src/gtest-assertion-result.o.d.raw   -c
g++: warning: switch '-msign-return-address=all' is no longer supported
  g++ -o /node/out/Release/obj.target/gtest/deps/googletest/src/gtest-death-test.o ../deps/googletest/src/gtest-death-test.cc '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_THREADS' '-DGTEST_HAS_POSIX_RE=0' '-DGTEST_LANG_CXX11=1' -I../deps/googletest -I../deps/googletest/include  -msign-return-address=all -pthread -Wall -Wextra -Wno-unused-parameter -fPIC -O3 -fno-omit-frame-pointer -fno-rtti -fno-exceptions -std=gnu++17 -MMD -MF /node/out/Release/.deps//node/out/Release/obj.target/gtest/deps/googletest/src/gtest-death-test.o.d.raw   -c
g++: warning: switch '-msign-return-address=all' is no longer supported
/tmp/ccNlzIJg.s: Assembler messages:
/tmp/ccNlzIJg.s:152: Error: selected processor does not support `pmull v20.1q,v1.1d,v0.1d'
/tmp/ccNlzIJg.s:156: Error: selected processor does not support `pmull v19.1q,v4.1d,v0.1d'
/tmp/ccNlzIJg.s:160: Error: selected processor does not support `pmull v18.1q,v3.1d,v0.1d'
/tmp/ccNlzIJg.s:164: Error: selected processor does not support `pmull v17.1q,v2.1d,v0.1d'
/tmp/ccNlzIJg.s:172: Error: selected processor does not support `pmull2 v1.1q,v1.2d,v0.2d'
/tmp/ccNlzIJg.s:176: Error: selected processor does not support `pmull2 v4.1q,v4.2d,v0.2d'
/tmp/ccNlzIJg.s:180: Error: selected processor does not support `pmull2 v3.1q,v3.2d,v0.2d'
/tmp/ccNlzIJg.s:184: Error: selected processor does not support `pmull2 v2.1q,v2.2d,v0.2d'
/tmp/ccNlzIJg.s:210: Error: selected processor does not support `pmull v0.1q,v1.1d,v5.1d'
/tmp/ccNlzIJg.s:214: Error: selected processor does not support `pmull2 v1.1q,v1.2d,v5.2d'
/tmp/ccNlzIJg.s:222: Error: selected processor does not support `pmull v1.1q,v0.1d,v5.1d'
/tmp/ccNlzIJg.s:226: Error: selected processor does not support `pmull2 v0.1q,v0.2d,v5.2d'
/tmp/ccNlzIJg.s:234: Error: selected processor does not support `pmull v0.1q,v1.1d,v5.1d'
/tmp/ccNlzIJg.s:238: Error: selected processor does not support `pmull2 v1.1q,v1.2d,v5.2d'
/tmp/ccNlzIJg.s:249: Error: selected processor does not support `pmull v2.1q,v0.1d,v5.1d'
/tmp/ccNlzIJg.s:253: Error: selected processor does not support `pmull2 v0.1q,v0.2d,v5.2d'
/tmp/ccNlzIJg.s:265: Error: selected processor does not support `pmull v2.1q,v0.1d,v5.1d'
/tmp/ccNlzIJg.s:269: Error: selected processor does not support `pmull2 v0.1q,v0.2d,v5.2d'
/tmp/ccNlzIJg.s:281: Error: selected processor does not support `pmull v2.1q,v0.1d,v5.1d'
/tmp/ccNlzIJg.s:285: Error: selected processor does not support `pmull2 v0.1q,v0.2d,v5.2d'
/tmp/ccNlzIJg.s:295: Error: selected processor does not support `pmull v5.1q,v0.1d,v5.1d'
/tmp/ccNlzIJg.s:309: Error: selected processor does not support `pmull v5.1q,v5.1d,v2.1d'
/tmp/ccNlzIJg.s:316: Error: selected processor does not support `pmull v2.1q,v2.1d,v4.1d'
/tmp/ccNlzIJg.s:327: Error: selected processor does not support `pmull v2.1q,v2.1d,v4.1d'
/tmp/ccNlzIJg.s:334: Error: selected processor does not support `pmull v1.1q,v1.1d,v3.1d'
make[1]: *** [deps/zlib/zlib_arm_crc32.target.mk:95: /node/out/Release/obj.target/zlib_arm_crc32/deps/zlib/crc32_simd.o] Error 1

This happens on both, GCC 11.3.0 and on GCC 7.5.0

[bnoordhuis]

It should check the host CPU for the available instructions

It does; see deps/zlib/cpu_features.c, the arm_cpu_enable_crc32 variable in particular.

Apropos gcc: it doesn't support setting target machine attributes on a per-function basis¹, only per compilation unit. If that's the problem you're hitting, try building with clang instead.

¹ Newer gcc versions possibly do. You can try submitting a fix to the upstream project; it's Chromium's zlib fork.

[JulianGro]

On an old GCC version, compilation failes with

  cc -o /node/out/Release/obj.target/zlib_arm_crc32/deps/zlib/crc32_simd.o ../deps/zlib/crc32_simd.c '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_THREADS' '-DCRC32_ARMV8_CRC32' '-DARMV8_OS_LINUX' -I../deps/zlib  -msign-return-address=all -pthread -Wall -Wextra -Wno-unused-parameter -fPIC -march=armv8-a+aes+crc -O3 -fno-omit-frame-pointer  -MMD -MF /node/out/Release/.deps//node/out/Release/obj.target/zlib_arm_crc32/deps/zlib/crc32_simd.o.d.raw   -c
cc1: error: invalid feature modifier in '-march=armv8-a+aes+crc'

on a Raspberry Pi 4, which suggests that it doesn't suggest instructions in practice.

The documentation lists gcc and g++ >= 10.1 or newer as build dependency, so it shouldn't depend on any features that gcc doesn't provide.

[bnoordhuis]

Yes, and it's building fine on our CI. The problem is likely local to your toolchain.

[bnoordhuis]

I'll close this out but let me know if there's still something to do or discuss.