.npmignores / reducing install bloat

[mattdesl]

A fresh install of browserify produces around 26.6 mb of bloat. Most of it is coming from tests, and tests of dependencies. Not a big deal for a global install, but when you are using browserify as a local dependency for WebGL/DOM testing across several dozens of modules, it starts to add up.

Some of the heavy hitters, sorted from lowest to highest in filesize:

384 shasum/
392 duplexer2/
416 resolve/
440 through2/
528 concat-stream/
560 buffer/
720 labeled-stream-splicer/
816 glob/
944 browser-resolve/
1176    JSONStream/
1272    browser-pack/
3144    umd/
5280    crypto-browserify/
5344    browserify-zlib/
5544    syntax-error/
7008    insert-module-globals/
12328   module-deps/

I could start submitting PRs for some of these, as they are easy changes.

[adam-lynch]

+1

cc @4ver

[sindresorhus]

👍

[kevva]

👍

[jeromedecoster]

👍

[sonicoder86]

start pushing the merge requests for this

[ghost]

Those sizes are in bytes:

$ curl -s https://registry.npmjs.org/module-deps/-/module-deps-3.7.2.tgz | wc -c
13747

This doesn't seem worth optimizing. In fact, if the biggest dependency is just 13 kilobytes, it sounds like we're doing really really well. I suspect the slowness of an install has more to do with the latency of round-trips to the registry for each dependency.

[ghost]

If the issue is size on disk after unpacking, 1) disk is pretty cheap 2) you can use symlinks if you've got lots of browserify copies of the same files floating around 3) you can find -name test -or -name example -exec rm -r {} \; after installing to save some room on disk.

[ghost]

I'm also not saying the slowness problem isn't real, but I think the best place to solve it will be in the npm client and registry to reduce the number of round-trips, since the size of the payload seems rather small.

[max-mapper]

I could be wrong, but I think this issue is more about stuff like acorn including it's test directory on npm.

For example, here's Disk Inventory X output on the browserify folder after a fresh npm install --save (e.g. no devDeps)

Here are the top offending modules in terms of on-disk usage:

edit forgot to add, my opinion is these large test files should get added to .npmignore in all the offending modules, but individual maintainers tend to have different philosophies in regards to publishing tests to npm

[ghost]

On-disk size is very misleading since everything under node_modules is included in that calculation. Over-the-wire sizes give a much better indication of package heft:

~ $ curl -s https://registry.npmjs.org/acorn/-/acorn-0.11.0.tgz | wc -c
423153
~ $ curl -s https://registry.npmjs.org/syntax-error/-/syntax-error-1.1.2.tgz | wc -c
3374
~ $ curl -s https://registry.npmjs.org/browserify-zlib/-/browserify-zlib-0.1.4.tgz | wc -c
64977

Ignoring the test directory on a 13k package seems pointless but slimming down acorn seems reasonable.

[max-mapper]

I just looked at my own dev computer, and I have over 200 copies of browserify installed taking up a total of over 4GB of space

Some are just folders called browserify, hence the 264 number, but I checked and all the folders included that arent actual copies of browserify take up only 8kb each.

[mattdesl]

looks like acorn will be ignoring tests soon
acornjs/acorn@2b6f24e

[zertosh]

@mattdesl Without the acorn tests the size should drop ~2MB acornjs/acorn#211. Also from browserify v8 to v9, you should've seen another ~1MB drop ForbesLindesay/umd#20