npm install fails without --build-from-source flag on node 12 on Ubuntu 18.04

[MasterOdin]

OS: Ubuntu 18.40
Node: 12.16.3
NPM: 6.14.4

Working in a directory with only dependency of sqlite: ^4.2.0, if I do npm install, it will always fail, while npm install --build-from-source always works.

The output of npm install is roughly:

mpeveler@Bishop:/mnt/e/Github/test$ npm install

> [email protected] install /mnt/e/Github/test/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using needle for node-pre-gyp https download
node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v72 ABI, glibc) (falling back to source compile with node-gyp)

node-pre-gyp WARN Hit error Remote end closed socket abruptly.
node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v72 ABI, glibc) (falling back to source compile with node-gyp)

node-pre-gyp WARN Hit error bad download
make: Entering directory '/mnt/e/Github/test/node_modules/sqlite3/build'
make: Warning: File 'node_sqlite3.target.mk' has modification time 1.3 s in the future
make: Entering directory '/mnt/e/Github/test/node_modules/sqlite3/build'
make: Warning: File 'node_sqlite3.target.mk' has modification time 1.3 s in the future
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3310100/sqlite3.c
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3310100/sqlite3.c
  TOUCH Release/obj.target/deps/action_before_build.stamp
  CC(target) Release/obj.target/sqlite3/gen/sqlite-autoconf-3310100/sqlite3.o
  TOUCH Release/obj.target/deps/action_before_build.stamp
  CC(target) Release/obj.target/sqlite3/gen/sqlite-autoconf-3310100/sqlite3.o

# bunch of build warnings

rm: cannot remove './Release/.deps/Release/obj.target/sqlite3/gen/sqlite-autoconf-3310100/sqlite3.o.d.raw': No such file or directory
deps/sqlite3.target.mk:143: recipe for target 'Release/obj.target/sqlite3/gen/sqlite-autoconf-3310100/sqlite3.o' failed
make: *** [Release/obj.target/sqlite3/gen/sqlite-autoconf-3310100/sqlite3.o] Error 1
make: Leaving directory '/mnt/e/Github/test/node_modules/sqlite3/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:310:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
  AR(target) Release/obj.target/deps/sqlite3.a
gyp ERR! System Linux 4.19.84-microsoft-standard
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64" "--napi_version=5" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v72"
gyp ERR! cwd /mnt/e/Github/test/node_modules/sqlite3
gyp ERR! node -v v12.16.3
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64 --napi_version=5 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/mnt/e/Github/test/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:310:20)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1021:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
node-pre-gyp ERR! System Linux 4.19.84-microsoft-standard
node-pre-gyp ERR! command "/usr/bin/node" "/mnt/e/Github/test/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /mnt/e/Github/test/node_modules/sqlite3
node-pre-gyp ERR! node -v v12.16.3
node-pre-gyp ERR! node-pre-gyp -v v0.11.0
node-pre-gyp ERR! not ok
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64 --napi_version=5 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v72' (1)
  COPY Release/sqlite3.anstall: info lifecycle [email protected]~install: Failed to exec install script
  CXX(target) Release/obj.target/node_sqlite3/src/backup.o.0~install: Failed to exec install script
../src/backup.cc:6:10: fatal error: macros.h: No such file or directory
compilation terminated.
node_sqlite3.target.mk:136: recipe for target 'Release/obj.target/node_sqlite3/src/backup.o' failed
make: *** [Release/obj.target/node_sqlite3/src/backup.o] Error 1
make: Leaving directory '/mnt/e/Github/test/node_modules/sqlite3/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:310:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
gyp ERR! System Linux 4.19.84-microsoft-standard
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/mnt/e/Github/test/node_modules/sqlite3/lib/binding/node-v72-linux-x64" "--napi_version=5" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v72"
gyp ERR! cwd /mnt/e/Github/test/node_modules/sqlite3
gyp ERR! node -v v12.16.3
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/mpeveler/.npm/_logs/2020-05-17T19_17_14_135Z-debug.log

The build warnings seem the same in both cases, while doing --build-from-source does give a slightly shorter header of messages before the build stuff:

npm install --build-from-source

> [email protected] install /mnt/e/Github/test/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using needle for node-pre-gyp https download
make: Entering directory '/mnt/e/Github/test/node_modules/sqlite3/build'
make: Warning: File 'node_sqlite3.target.mk' has modification time 1.3 s in the future
  ACTION deps_sqlite3_gyp_action_before_build_target_unpack_sqlite_dep Release/obj/gen/sqlite-autoconf-3310100/sqlite3.c
  TOUCH Release/obj.target/deps/action_before_build.stamp
  CC(target) Release/obj.target/sqlite3/gen/sqlite-autoconf-3310100/sqlite3.o

[MasterOdin]

Alright, looks like node-pre-gyp has a bug where it can trigger node-gyp twice, which is causing this build corruption. Upgrading node-pre-gyp to 0.12+ fixes the issue. #1265 will fix this, but only for the 5.x line. Would you be willing to make a 4.2.1 release with just node-pre-gyp updated as I'm not ready to jump to v5 and drop all versions of node < 11.

@KWEDE I'm happy to do any and all legwork, testing, etc. involved in making this happen.

[kewde]

034c05f

[daniellockyer]

Would you be able to try on the latest version - v5.0.3? 🙂

[MasterOdin]

Yeah, 5.0.3 works fine since it brings in a newer version of node-pre-gyp.