Skip to content

new core modules go under a namespace #21551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
2 changes: 1 addition & 1 deletion benchmark/fixtures/echo.worker.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
'use strict';

const { parentPort } = require('worker_threads');
const { parentPort } = require('@nodejs/worker_threads');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm strongly -1 on renaming worker_threads at this point, fwiw


parentPort.on('message', (msg) => {
parentPort.postMessage(msg);
Expand Down
2 changes: 1 addition & 1 deletion benchmark/fixtures/require-cachable.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
const list = require('internal/bootstrap/cache');
const {
isMainThread
} = require('worker_threads');
} = require('@nodejs/worker_threads');

for (const key of list.cachableBuiltins) {
if (!isMainThread && key === 'trace_events') {
Expand Down
2 changes: 1 addition & 1 deletion benchmark/misc/startup.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ function main({ dur, script, mode }) {

script = path.resolve(__dirname, '../../', `${script}.js`);
if (mode === 'worker') {
Worker = require('worker_threads').Worker;
Worker = require('@nodejs/worker_threads').Worker;
bench.start();
start(state, script, bench, spawnWorker);
} else {
Expand Down
2 changes: 1 addition & 1 deletion benchmark/worker/echo.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ const bench = common.createBenchmark(main, {
const workerPath = path.resolve(__dirname, '..', 'fixtures', 'echo.worker.js');

function main(conf) {
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');

const n = +conf.n;
const workers = +conf.workers;
Expand Down
6 changes: 6 additions & 0 deletions doc/api/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -1234,6 +1234,12 @@ An invalid HTTP token was supplied.

An IP address is not valid.

<a id="ERR_INVALID_NODE_BUILTIN"></a>
### ERR_INVALID_NODE_BUILTIN

An attempt was made to load a module from the Node.js namespace that doesn't
exist. Only built in modules can be require from '@nodejs/'.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
exist. Only built in modules can be require from '@nodejs/'.
exist. Only built-in modules can be required from '@nodejs/'.


<a id="ERR_INVALID_OPT_VALUE"></a>
### ERR_INVALID_OPT_VALUE

Expand Down
40 changes: 21 additions & 19 deletions doc/api/worker_threads.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on independent threads, and to create message channels between them. It
can be accessed using the `--experimental-worker` flag and:

```js
const worker = require('worker_threads');
const worker = require('@nodejs/worker_threads');
```

Workers are useful for performing CPU-intensive JavaScript operations; do not
Expand All @@ -23,7 +23,7 @@ share memory efficiently by transferring `ArrayBuffer` instances or sharing
```js
const {
Worker, isMainThread, parentPort, workerData
} = require('worker_threads');
} = require('@nodejs/worker_threads');

if (isMainThread) {
module.exports = async function parseJSAsync(script) {
Expand Down Expand Up @@ -104,7 +104,7 @@ yields an object with `port1` and `port2` properties, which refer to linked
[`MessagePort`][] instances.

```js
const { MessageChannel } = require('worker_threads');
const { MessageChannel } = require('@nodejs/worker_threads');

const { port1, port2 } = new MessageChannel();
port1.on('message', (message) => console.log('received', message));
Expand Down Expand Up @@ -243,8 +243,10 @@ Notable differences inside a Worker environment are:

- The [`process.stdin`][], [`process.stdout`][] and [`process.stderr`][]
may be redirected by the parent thread.
- The [`require('worker_threads').isMainThread`][] property is set to `false`.
- The [`require('worker_threads').parentPort`][] message port is available,
- The [`require('@nodejs/worker_threads').isMainThread`][]
property is set to `false`.
- The [`require('@nodejs/worker_threads').parentPort`][] message port is
available,
- [`process.exit()`][] does not stop the whole program, just the single thread,
and [`process.abort()`][] is not available.
- [`process.chdir()`][] and `process` methods that set group or user ids
Expand Down Expand Up @@ -286,7 +288,7 @@ the thread barrier.
const assert = require('assert');
const {
Worker, MessageChannel, MessagePort, isMainThread, parentPort
} = require('worker_threads');
} = require('@nodejs/worker_threads');
if (isMainThread) {
const worker = new Worker(__filename);
const subChannel = new MessageChannel();
Expand Down Expand Up @@ -314,10 +316,10 @@ if (isMainThread) {
* `eval` {boolean} If true, interpret the first argument to the constructor
as a script that is executed once the worker is online.
* `workerData` {any} Any JavaScript value that will be cloned and made
available as [`require('worker_threads').workerData`][]. The cloning will
occur as described in the [HTML structured clone algorithm][], and an error
will be thrown if the object cannot be cloned (e.g. because it contains
`function`s).
available as [`require('@nodejs/worker_threads').workerData`][]. The
cloning will occur as described in the [HTML structured clone algorithm][],
and an error will be thrown if the object cannot be cloned (e.g. because it
contains `function`s).
* stdin {boolean} If this is set to `true`, then `worker.stdin` will
provide a writable stream whose contents will appear as `process.stdin`
inside the Worker. By default, no data is provided.
Expand Down Expand Up @@ -356,7 +358,7 @@ added: v10.5.0
* `value` {any} The transmitted value

The `'message'` event is emitted when the worker thread has invoked
[`require('worker_threads').postMessage()`][]. See the [`port.on('message')`][]
[`require('@nodejs/worker_threads').postMessage()`][]. See the [`port.on('message')`][]
event for more details.

### Event: 'online'
Expand All @@ -376,7 +378,7 @@ added: v10.5.0
* `transferList` {Object[]}

Send a message to the worker that will be received via
[`require('worker_threads').parentPort.on('message')`][].
[`require('@nodejs/worker_threads').parentPort.on('message')`][].
See [`port.postMessage()`][] for more details.

### worker.ref()
Expand Down Expand Up @@ -451,7 +453,7 @@ added: v10.5.0
* {integer}

An integer identifier for the referenced thread. Inside the worker thread,
it is available as [`require('worker_threads').threadId`][].
it is available as [`require('@nodejs/worker_threads').threadId`][].

### worker.unref()
<!-- YAML
Expand Down Expand Up @@ -480,12 +482,12 @@ active handle in the event system. If the worker is already `unref()`ed calling
[`process.stdin`]: process.html#process_process_stdin
[`process.stdout`]: process.html#process_process_stdout
[`process.title`]: process.html#process_process_title
[`require('worker_threads').isMainThread`]: #worker_threads_worker_ismainthread
[`require('worker_threads').parentPort.on('message')`]: #worker_threads_event_message
[`require('worker_threads').parentPort`]: #worker_threads_worker_parentport
[`require('worker_threads').postMessage()`]: #worker_threads_worker_postmessage_value_transferlist
[`require('worker_threads').threadId`]: #worker_threads_worker_threadid
[`require('worker_threads').workerData`]: #worker_threads_worker_workerdata
[`require('@nodejs/worker_threads').isMainThread`]: #worker_threads_worker_ismainthread
[`require('@nodejs/worker_threads').parentPort.on('message')`]: #worker_threads_event_message
[`require('@nodejs/worker_threads').parentPort`]: #worker_threads_worker_parentport
[`require('@nodejs/worker_threads').postMessage()`]: #worker_threads_worker_postmessage_value_transferlist
[`require('@nodejs/worker_threads').threadId`]: #worker_threads_worker_threadid
[`require('@nodejs/worker_threads').workerData`]: #worker_threads_worker_workerdata
[`trace_events`]: tracing.html
[`worker.on('message')`]: #worker_threads_event_message_1
[`worker.postMessage()`]: #worker_threads_worker_postmessage_value_transferlist
Expand Down
3 changes: 2 additions & 1 deletion lib/internal/bootstrap/loaders.js
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,8 @@ if (config.exposeInternals) {

NativeModule.isInternal = function(id) {
return id.startsWith('internal/') ||
(id === 'worker_threads' && !config.experimentalWorker);
((id === '@nodejs/worker_threads' || id === 'worker_threads') &&
!config.experimentalWorker);
};
}

Expand Down
1 change: 1 addition & 0 deletions lib/internal/errors.js
Original file line number Diff line number Diff line change
Expand Up @@ -749,6 +749,7 @@ E('ERR_INVALID_FILE_URL_PATH', 'File URL path %s', TypeError);
E('ERR_INVALID_HANDLE_TYPE', 'This handle type cannot be sent', TypeError);
E('ERR_INVALID_HTTP_TOKEN', '%s must be a valid HTTP token ["%s"]', TypeError);
E('ERR_INVALID_IP_ADDRESS', 'Invalid IP address: %s', TypeError);
E('ERR_INVALID_NODE_BUILTIN', '%s is not a builtin Node.js module', Error);
E('ERR_INVALID_OPT_VALUE', (name, value) =>
`The value "${String(value)}" is invalid for option "${name}"`,
TypeError,
Expand Down
20 changes: 13 additions & 7 deletions lib/internal/modules/cjs/helpers.js
Original file line number Diff line number Diff line change
Expand Up @@ -95,24 +95,28 @@ function stripShebang(content) {
return content;
}

const nonNamespaceBlacklist = new Set();

if (getOptionValue('--experimental-worker')) {
nonNamespaceBlacklist.add('worker_threads');
}

const builtinLibs = [
'assert', 'async_hooks', 'buffer', 'child_process', 'cluster', 'crypto',
'dgram', 'dns', 'domain', 'events', 'fs', 'http', 'http2', 'https', 'net',
'os', 'path', 'perf_hooks', 'punycode', 'querystring', 'readline', 'repl',
'stream', 'string_decoder', 'tls', 'trace_events', 'tty', 'url', 'util',
'v8', 'vm', 'zlib'
'v8', 'vm', 'zlib', ...nonNamespaceBlacklist
];

if (getOptionValue('--experimental-worker')) {
builtinLibs.push('worker_threads');
builtinLibs.sort();
}

if (typeof process.binding('inspector').open === 'function') {
builtinLibs.push('inspector');
builtinLibs.sort();
}

builtinLibs.sort();

const namespaceWhitelist = new Set(builtinLibs);

function addBuiltinLibsToObject(object) {
// Make built-in modules available directly (loaded lazily).
builtinLibs.forEach((name) => {
Expand Down Expand Up @@ -156,6 +160,8 @@ module.exports = exports = {
addBuiltinLibsToObject,
builtinLibs,
makeRequireFunction,
namespaceWhitelist,
nonNamespaceBlacklist,
requireDepth: 0,
stripBOM,
stripShebang
Expand Down
14 changes: 13 additions & 1 deletion lib/internal/modules/cjs/loader.js
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ const {
const { safeGetenv } = internalBinding('util');
const {
makeRequireFunction,
namespaceWhitelist,
nonNamespaceBlacklist,
requireDepth,
stripBOM,
stripShebang
Expand All @@ -48,6 +50,7 @@ const experimentalModules = getOptionValue('--experimental-modules');

const {
ERR_INVALID_ARG_VALUE,
ERR_INVALID_NODE_BUILTIN,
ERR_REQUIRE_ESM
} = require('internal/errors').codes;
const { validateString } = require('internal/validators');
Expand Down Expand Up @@ -567,7 +570,16 @@ function tryModuleLoad(module, filename) {
}

Module._resolveFilename = function(request, parent, isMain, options) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To the best of your knowledge, is there any popular module monkeypatching this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could make this a semver-major. As @mcollina suggests, we need to determine if this breaks anyone. Likely candidates are APM or testing libraries.

This comment was marked as outdated.

This comment was marked as outdated.

if (NativeModule.nonInternalExists(request)) {
Copy link
Member

@joyeecheung joyeecheung Jan 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-post as this seems to be buried from the timeline:

Maybe the correct place to insert this logic is NativeModule.nonInternalExists instead of Module._resolveFilename, also NativeModule.nonInternalExists is even less public anyways (you'd need --expose-internals to get access to NativeModule). You can change it to something like NativeModule.getUnifiedId which returns undefined if there is no such module with or without the prefix. That would also fix up Module._resolveLookupPaths.

It could also be faster if NativeModule.getUnifiedId simply queries a map of key => { unified_id, source }with both @nodejs/something and something inserted as keys pointing to the same entry. Right now NativeModule.nonInternalExists does NativeModule._source.hasOwnProperty(id) in the end but NativeModule._source is entirely internal so we are free to change its structure. That'll avoid the extra overhead brought by the blacklist and whitelist mentioned in the other review as there is still only one lookup, just in a bigger map (BTW this object is already in dictionary mode so I guess using JS Map would be faster).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to update this, where would I find the source for NativeModule?

if (request.startsWith('@nodejs/')) {
request = request.slice(8);
if (!namespaceWhitelist.has(request)) {
throw new ERR_INVALID_NODE_BUILTIN(request);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a test to verify this case? I haven't found one in this PR. This is a very important feature that prevents misusage.

}
return request;
}

if (!nonNamespaceBlacklist.has(request) &&
NativeModule.nonInternalExists(request)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to benchmark this change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a suggestion on how to do this or an exiting benchmark we can run?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible a new benchmark would need to be written. We need to know the impact here

return request;
}

Expand Down
2 changes: 1 addition & 1 deletion test/abort/test-addon-uv-handle-leak.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ const assert = require('assert');
const fs = require('fs');
const path = require('path');
const cp = require('child_process');
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');
const { spawnSync } = require('child_process');

// This is a sibling test to test/addons/uv-handle-leak.
Expand Down
2 changes: 1 addition & 1 deletion test/addons/uv-handle-leak/test.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ const binding = require(bindingPath);
try {
// We don’t want to run this in Workers because they do actually enforce
// a clean-exit policy.
const { isMainThread } = require('worker_threads');
const { isMainThread } = require('@nodejs/worker_threads');
if (!isMainThread)
common.skip('Cannot run test in environment with clean-exit policy');
} catch {}
Expand Down
2 changes: 1 addition & 1 deletion test/common/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ const noop = () => {};

const isMainThread = (() => {
try {
return require('worker_threads').isMainThread;
return require('@nodejs/worker_threads').isMainThread;
} catch {
// Worker module not enabled → only a single main thread exists.
return true;
Expand Down
2 changes: 1 addition & 1 deletion test/common/wpt.js
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ class WPTRunner {
// TODO(joyeecheung): we are not a window - work with the upstream to
// add a new scope for us.

const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');
sandbox.DedicatedWorker = Worker; // Pretend we are a Worker
return context;
}
Expand Down
2 changes: 1 addition & 1 deletion test/fixtures/v8-coverage/worker.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
'use strict';
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');
const path = require('path');

new Worker(path.resolve(__dirname, 'subprocess.js'));
2 changes: 1 addition & 1 deletion test/parallel/test-async-wrap-missing-method.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
const common = require('../common');
const assert = require('assert');

const { MessageChannel } = require('worker_threads');
const { MessageChannel } = require('@nodejs/worker_threads');

{
const { port1, port2 } = new MessageChannel();
Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-heapdump-worker.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
'use strict';
require('../common');
const { validateSnapshotNodes } = require('../common/heap');
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');

validateSnapshotNodes('Node / Worker', []);
const worker = new Worker('setInterval(() => {}, 100);', { eval: true });
Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-trace-events-api-worker-disabled.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
'use strict';

const common = require('../common');
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');

new Worker("require('trace_events')", { eval: true })
.on('error', common.expectsError({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
'use strict';

const common = require('../common');
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');

common.skipIfInspectorDisabled();

Expand Down
4 changes: 2 additions & 2 deletions test/parallel/test-trace-events-worker-metadata.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ const common = require('../common');
const assert = require('assert');
const cp = require('child_process');
const fs = require('fs');
const { isMainThread } = require('worker_threads');
const { isMainThread } = require('@nodejs/worker_threads');

if (isMainThread) {
const CODE = 'const { Worker } = require(\'worker_threads\'); ' +
const CODE = 'const { Worker } = require(\'@nodejs/worker_threads\'); ' +
`new Worker('${__filename.replace(/\\/g, '/')}')`;
const FILE_NAME = 'node_trace.1.log';
const tmpdir = require('../common/tmpdir');
Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-worker-cleanup-handles.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
'use strict';
const common = require('../common');
const assert = require('assert');
const { Worker, isMainThread, parentPort } = require('worker_threads');
const { Worker, isMainThread, parentPort } = require('@nodejs/worker_threads');
const { Server } = require('net');
const fs = require('fs');

Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-worker-debug.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ const { Session } = require('inspector');
const { pathToFileURL } = require('url');
const {
Worker, isMainThread, parentPort, workerData
} = require('worker_threads');
} = require('@nodejs/worker_threads');


const workerMessage = 'This is a message from a worker';
Expand Down
4 changes: 2 additions & 2 deletions test/parallel/test-worker-dns-terminate.js
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
// Flags: --experimental-worker
'use strict';
const common = require('../common');
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');

const w = new Worker(`
const dns = require('dns');
dns.lookup('nonexistent.org', () => {});
require('worker_threads').parentPort.postMessage('0');
require('@nodejs/worker_threads').parentPort.postMessage('0');
`, { eval: true });

w.on('message', common.mustCall(() => {
Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-worker-esmodule.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
const common = require('../common');
const fixtures = require('../common/fixtures');
const assert = require('assert');
const { Worker } = require('worker_threads');
const { Worker } = require('@nodejs/worker_threads');

const w = new Worker(fixtures.path('worker-script.mjs'));
w.on('message', common.mustCall((message) => {
Expand Down
2 changes: 1 addition & 1 deletion test/parallel/test-worker-exit-code.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ const common = require('../common');
// in multiple situations.

const assert = require('assert');
const worker = require('worker_threads');
const worker = require('@nodejs/worker_threads');
const { Worker, parentPort } = worker;

const testCases = require('../fixtures/process-exit-code-cases');
Expand Down
Loading