[RRFC] Add flag for running NPM commands in transitive dependencies

[zackerydev]

Motivation ("The Why")

There are many tools to manage multiple NPM packages in a concise and user friendly way. For a long time the front runner for those were Lerna and Yarn Workspaces.

NPM mono repos tools usually have two main features:
- Installing node_modules across an entire tree of dependencies and symlinking any dependencies in the tree that are found to be local to the mono repo
- Running commands across those packages

With the release of NPM 7 and NPM workspace npm itself came much closer to having the features needed out of the box to run a mono repo. This was great because Lerna has since been largely abandoned and the Yarn ecosystem was fragmented with the release of v2. However, NPM workspaces is lacking in the ability to run commands across transitive dependencies.

There are other mono repo tools gaining popularity like TurboRepo and NX but those have their own toolchains and features to consider, I think it would be preferred for NPM to support this out of the box.

It is important to note that this feature was expected to function this way by enough users that two issues npm/cli#3413 and npm/cli#4139 both considered this a bug and not a new feature.

Example

Having this feature would allow for mono repo tooling to run exclusively with npm something that comes out of the box with Node.js. Not requiring any other toolchains or special workflows.

How

Current Behaviour

Take this repo for example:
https://github.com/zgriesinger/monorepo

In it we have a npm workspace with frontends, apis, cli tools and shared packages.

If I pull the app down and:

npm i
npm run build -w @zgriesinger/service-a

I'll be met with errors since the @zgriesinger/logger package has not been built (but it has been symlinked)

If I know the transitive dependencies of service-a that are local to the repo I can do this to get the build working

npm run build -w @zgriesinger/logger
npm run build -w @zgriesinger/service-a

This is why most mono repos have to keep some other toolchain around, with lerna there is the --include-dependencies flag that will automatically run the build for @zgriesinger/logger

Desired Behaviour

The desired behavior of the npm cli would be to include a --include-dependencies flag similar to lerna.

The functionality/algorithm of this flag would be as follows:

• Read dependencies of targeted workspace with -w flag
  • If dependencies includes any internal to the mono repo, read them and start from the beginning (Including matching versions)
  • If not, run the desired npm run script in that package, and return up to run the command in the parent package

The result should be in the example above:

npm run build -w @zgriesinger/service-a
  -> Runs build in @zgriesinger/logger
  -> On succes runs build in @zgriesinger/service-a

References

• [BUG] run transitive dependencies with --workspace flag? cli#3413
• [BUG] workspace scripts should be run in topological order by default cli#4139

[bnb]

+1 to this behind a flag.

[ljharb]

Per discussion, this seems analogous to lerna's --include-dependencies flag.

A separate RFC would perhaps be needed (or, maybe it's just considered a bug) that if A depends on B, and i explicitly npm run whatever -w a -w b, that npm should just "figure out" that B's script needs to run before A's script does, instead of in parallel.

[zackerydev]

Linking for Prior Art:
https://turborepo.org/blog/turbo-1-2-0#new-task-filtering-api

I like this api much more than Lerna's --include-dependencies flag but the include dependencies paradigm might be more familiar for existing lerna users.

[diegohaz]

A separate RFC would perhaps be needed (or, maybe it's just considered a bug) that if A depends on B, and i explicitly npm run whatever -w a -w b, that npm should just "figure out" that B's script needs to run before A's script does, instead of in parallel.

Yeah! I'm not sure this RFC covers npm/cli#4139. But #442 was closed in favor of this, so I guess that should be part of it.

[darcyclarke]

Leaving this on the agenda again for today & also just wanted to reference wireit (ref. https://github.com/google/wireit)