module: increased surface for hazards with require(esm) experimental flag

[WebReflection]

What is the problem this feature will solve?

In this MR #51977 it's being proposed to allow CJS to require(esm) with the goal of helping people stuck in CJS to use ESM only modules, somehow conflicting with the plethora of dual modules already published and maintained by authors (and I am one of them).

To understand the issue there's no better way than a concrete use case.

---

Where we are now ...

A project using module a as dependency, where a is published as dual module, can be consumed from pure CJS, where by pure I mean no dynamic import(a) in the mix, as that usually undesired or not common in CJS land due lack of TLA, as well as pure ESM.

module a

// a.mjs
export default Math.random();

// a.cjs
module.exports = Math.random();

module b

console.log(require("a")); // ./a.cjs
// 0.123456789

In this CJS scenario the module a will always provide the same random number once, no matter how many modules require it.

Now, because it was not possible before to synchronously import module c, the module b has no hazards in a CJS only environment.

Now enters module c as ./c.mjs

// module c depends on module `a`
// and because it's ESM only it will
// consume module `a` as ESM
import random from "a"; // ./a.mjs

console.log(random);
// 0.234567891

export default random * 2;

Where we're potentially going ...

edit test the use case if you want

In a scenario where the flag lands "unflagged" and the CJS is still the default, developers will carelessly believe they can finally require(esm) without thinking twice about possible consequences ... if no error is encountered due TLA in the required ESM module, they think they're good!

module b after the flag

console.log(require("a")); // ./a.cjs
// 0.123456789

console.log(require("c").default); // ./c.mjs -> ./a.mjs
// NOT 0.246913578
// BUT 0.469135782

Conclusion

It is true that hazards related to dual modules where already possible before but there was no easy way to synchronously require ESM modules so that either they chose dual modules, they used a tool able to normalize everything as CJS or ESM, but any re-published package as CJS would've avoided or inlined somehow ESM modules with TLA.

In short, if the default require, when it comes to dual modules, still prefers CJS once this flag lands unflagged, we will see dual module authors blamed for issues that were not so easy to bring in before with potentially catastrophic results beyond the dummy Math.random() use case: databases, cache, file system, workers, you name it ... bootstrapping with ease dual modules will likely create more damage than solve instead anything it's aiming to solve and currently published dual modules cannot just stop providing their CJS counterpart until all users and dependents modules are capable of requiring ESM out of the box.

What is the feature you are proposing to solve the problem?

Before proposing anything I don't understand how this experimental flag is being considered as shippable in its current state (broken ESM require if TLA is used behind, increased dual module hazards surface) instead of fixing at the root level the issue by:

• allow ESM as default in NodeJS
• allow TLA in CJS too so that dynamic imports won't scare anyone anymore and the TLA behind the scene would also just work

Back to this flag though, I would like to propose at least the following solutions:

• the default require(anything), once this flag lands unflagged, is to return the ESM version of the module, if such module is a dual module. This would solve the presented use case / issue because the module b that require("a") will already have the ESM version so that once module b finally can also require("c") from the ESM only world, nothing will break and no hazard will be present
• there's gonna be a flag to impose the preferred way to require a module from CJS so that dual module authors can explicitly indicate in their package.json that the preferred way to require that module is through its ESM code and not its CJS artifact (or vice-versa whenever that's the case). In this case I believe having the "type": "module" in the package.json of a SHOULD already be enough to tell the new require(esm) ability that even when require(cjs) was used, and that module is dual module, the returned module is actually the ESM one.

The latter point was breaking before so it should be a no-brainer to consider while the former suggestion might break with default exports that were not expected before but I am hear to discuss possible solutions there too or propose we give developers time to test and adjust their code, after all they need to do so anyway the moment they finally require(ESM).

Having ESM as preferred way to any require will eventually convince developers to publish ESM only modules so that the dual module story can fade away in time, but if that's not the case I see a catch 22 like situation where authors can't stop publishing dual modules and users can't stop worrying about possible hazards introduced by the new feature.

What alternatives have you considered?

As already mentioned, I would rather bring in TLA in CJS behind a flag instead and call it a day, without mixing a tad too much the require ability of the legacy, non standard, module system that CJS is. This would be the easiest migration for everyone caring to migrate and it will still grant pure CJS software to work without worrying about hazards in the graph.

Thank you.

---

After Thoughts

It's OK to explore solutions to the problem (we're doing that already) but if NodeJS lands these kind of changes it better be sure that the entirety of the tooling based projects out there also get the memo and implement the right thing or we'll create a new kind of hell to deal with, explain, and resolve ... at least until everything is ESM only and CJS can be just a memory to find in outdated books.

[nicolo-ribaudo]

(thanks for moving the discussion from Twitter to GitHub!)

the default require(anything), once this flag lands unflagged, is to return the ESM version of the module, if such module is a dual module.

When Joyee tried to make require() accept the import condition rather than just require(), we found that it breaks one of the top-30 ESM packages that she tested (#51977 (comment)).

I'm a maintainer of that package so I could easily fix that, but the problem boils down to the require and import versions not having exactly the same API: the CJS version has a __esModule: true export while the ESM version doesn't. #52166 would fix it, so maybe it's worth considering again the breaking change.

Another case where you would not want require() to respect the import condition is when you have a sync CJS entrypoint and an async ESM entrypoint (for example, because they both do test ? require("./variant-a") : require("./variant-b") and test ? await import("./variant-a") : await import("./variant-b")). In this case, if require() resolved to the ESM version it would then start throwing while loading it. Again, there is a possible solution for package authors: given that conditions are matched in the order they are defined in package.json, you could choose if you want to give priority to import or require:

Priority to CJS	Priority to ESM
{ "name": "foo", "exports": { "require": "./foo.cjs", "import": "./foo.mjs" } }	{ "name": "foo", "exports": {, "import": "./foo.mjs", "require": "./foo.cjs" } }

This ability to choose priority applies both in the case of just extending require to also look at the import condition, and in the case of just adding a new condition such as require-esm.

---

Now, because it was not possible before to synchronously import module c, the module b has no hazards in a CJS only environment.

The require(esm) change does not affect CJS-only environments, since it's about mixing CJS and ESM. In mixed environments, such as the example you are giving, the hazard is already present using only synchronous code: import ... from declarations and require() are enough to trigger it (see the example in #52174).

Having ESM as preferred way to any require will eventually convince developers to publish ESM only modules so that the dual module story can fade away in time

Regardless of require() resolution priority, dual modules can be dropped exactly in the moment all the Node.js versions you care about support require(esm), right?

[WebReflection]

given that conditions are matched in the order they are defined in package.json, you could choose if you want to give priority to import or require

is this a theoretical or that's actually implemented in that flag already? I see your links are still open here and there

In mixed environments, such as the example you are giving, the hazard is already present using only synchronous code

there's no hazard if the first require("a") points at ./a.mjs instead ... dual module per se is not an hazard, is the hybrid env that imports here and there stuff differently that causes the hazard ... here I am suggesting to pivot to ESM always on require whenever that's possible. The change / order of the explicit import works though, I still need to update tons of modules but that's already something, if it works the way you suggested.

dual modules can be dropped exactly in the moment all the Node.js versions you care about support require(esm), right?

it'll take years, I suppose, but yes. New modules won't be published as dual, old modules with hundred million weekly downloads aren't so "easy-peasy" to change.

[nicolo-ribaudo]

is this a theoretical or that's actually implemented in that flag already? I see your links are still open here and there

It's reality that export conditions are matched in order, it's theoretical that require() could match the import condition.

there's no hazard if the first require("a") points at ./a.mjs instead

I agree, but that's not what already happens today without the flag, so today we are already affected by the problem even without using dynamic import (again, see the docs issue I linked because it contains an example :) )

[WebReflection]

@nicolo-ribaudo this might be a hell of a lucky coincidence but thanks to the fact I use npx modulestrap to bootstrap any project of mine it looks like all my dual packages are already like this:

  "exports": {
    ".": {
      "types": "./types/index.d.ts",
      "import": "./esm/index.js",
      "default": "./cjs/index.js"
    },
    "./package.json": "./package.json"
  },

Am I correct in understanding you that this means there's literally nothing I should do to change because require("flatted"), as example, or require("linkedom") or others will automatically prefer to import the ESM version, as that's first (well, after types, eventually, but that's not a NodeJS affair) in the package export definition?

I might rest my case if that's the case, still if this is how that flag lands everyone should be aware of this potential issue and be sure modules authors also understand how to solve it ... they have time to just eventually swap export definitions without breaking anything in the meantime, the breaking will eventually land once that flag becomes the default.

[nicolo-ribaudo]

If require matched also the import condition then yes, you would would have nothing to change because import would be the first matched condition in your list.

[WebReflection]

so, the current state is that given this exports definition in the module package that is being either imported or required:

{
  "name": "whatever",
  "type": "module",
  "exports": {
    ".": {
      "types": "./types/index.d.ts",
      "import": "./esm/index.js",
      "default": "./cjs/index.js"
    },
    "./package.json": "./package.json"
  }
}

once this flag gets unflagged the only file that will ever be required or imported is ./esm/index.js ... right?

edit explicitly:

// ./cjs/index.js -> exports.ref = {};
// with flag this points at ./esm/index.js instead
const { ref: cjs } = require("whatever");

// ./esm/index.js -> export const ref = {};
import { ref as esm } from "whatever";

// nothing to see here, it's all good!
console.assert(cjs === esm, "hazards in the house");

If this is the case then my proposal about signaling from module authors what should be preferred is already satisfied ... I just want to be sure this is the case so that at least I don't need to worry about this breaking change in the near future, thank you.

[nicolo-ribaudo]

No, the behavior of that package doesn't currently change under the flag.

What I was proposing above is to instead explore make the flag break backwards compatibility, so that in your example both require and import would match the import condition and that console.assert (that currently, even without this flag, fails) would pass.

[WebReflection]

that currently, even without this flag, fails

of course it does ... that was the whole point, I want it to not fail and force require("whatever") to point at ./esm/index.js instead ... can we agree module authors should have this ability? If not, this flag is going to be hostile for users, they add hazards without understanding what's going on, and for module authors, they get the blame for something they can't even control within the Open Source software they provide.

I hope we agree here that make authors able to chose what "future require" should prefer must be a requirement for this flag.

[WebReflection]

btw ...

that currently, even without this flag, fails

if your point is that it should have a createRequire upfront it's fine ... I wanted to meta describe the desired result from all worlds, no matter if that whatever module is required or imported ... I hope that clarifies.

[nicolo-ribaudo]

if your point is that it should have a createRequire upfront it's fine

No, my point is that today, in sync-only-code, there is no guarantee that a dual package that uses import or require conditions will be loaded only in one of the two versions. A library depending on yours might be written in CJS and load the CJS version; a library depending on yours might be written in ESM and load the ESM version; and the final users importing those two libraries will see both versions of yours being loaded.

---

can we agree module authors should have this ability?

Yes, I agree! I disagree that this is a breaking change introduced by the flag, but I agree that module authours should have this ability. And I see four concrete ways to do so, I'll list them here.

Do nothing

This might seem counter intuitive, but I just realized that even if Node.js does nothing you can still publish packages that have the behavior you want.

You need to prefix your transpiled CJS files with a require() of the ESM file:

package.json	foo.mjs	foo.cjs
{ "name": "foo", "exports": { "import": "./foo.mjs", "default": "./foo.cjs" } }	let num = Math.random(); export { num };	try { return require("./foo.mjs") } catch {} let num = Math.random(); exports.num = num;

Add a new module-sync condition that is matched both by import and require, in platforms where require() can load sync ESM

This would let you have different entrypoints for ESM with TLA (only loaded by import), ESM without TLA (loaded both by import and require, and CJS.

{
  "name": "foo",
  "exports": {
    "module-sync": "./foo.mjs",
    "default": "./foo.cjs"
  }
}

let num = Math.random();
export { num };

let num = Math.random();
exports.num = num;

Add a new require-esm condition that is matched by require in platforms where require() can load sync ESM

{
  "name": "foo",
  "exports": {
	"import": "./foo.mjs",
    "require-esm": "./foo.mjs",
    "default": "./foo.cjs"
  }
}

let num = Math.random();
export { num };

let num = Math.random();
exports.num = num;

Change require() to also match the "import" condition

This is a breaking change, but maybe it could be explored.

[WebReflection]

P.S. I wouldn't mind a situation like this neither, as ordered JSON is also something not many care about in the wild (AFAIK)

{
  "name": "whatever",
  "type": "module",
  "exports": {
    ".": {
      // edit: oops, taken, I meant require-esm
      "require": "./esm/index.js",
      "types": "./types/index.d.ts",
      "import": "./esm/index.js",
      "default": "./cjs/index.js"
    },
    "./package.json": "./package.json"
  }
}

This adds a new field, it's "scoped" per each defined export, and extremely easy to reason about to me ... we can debate if "cjs" instead of "require-esm" is a better choice, or literally any other convention to satisfy the goal, but I hope something happens that allows authors to decide themselves what's the best qay to require their own modules.

In my case, when no hazards or side effects are possible (i.e. just utilities and who cares if duplicated), I could decide myself which path should be served and prefer maybe CJS if there's only a default export, translated directly as module.exports = thing without all the ugly __esmModule and oter decorations too many tools add these days to solve the same issue.

This would probably be my best option yet because it's also easier for the eyes to understand, and easier for everyone to reason about, imho.

[nicolo-ribaudo]

That package.json is incompatible with every Node.js version between 12 and <last-version-without-the-flag-enabled-by-default>.

EDIT Oh yes, with a new name it's like the require-esm I proposed above.