feat(compartment-mapper): support mapping of undiscoverable packages

[boneskull]

Description

This adds options to mapNodeModules(), captureFromMap(), loadFromMap() and importLocation() to support loading additional packages that wouldn't otherwise be discovered by mapNodeModules(). It takes several ideas from #2864 which were not already implemented in #2876.

The first option is additionalModuleLocations, which is an Array<string | AdditionalModuleLocationObject> where items can either be fully-qualified file:// URLs or an object with shape:

export interface AdditionalModuleLocationObject {
  location: string;
  dev?: boolean;
  conditions?: Set<string>;
}

Using the above object, consumers have granular control over certain options when graphing additional modules. If dev or conditions are provided ("not undefined") as options to mapNodeModules() directly, the additional modules will inherit those settings (there is coverage for this).

The second option, additionalPackageDetails, is a superset of AdditionalModuleLocationObject which also contains a package descriptor and package location. This array can then be handed to captureFromMap() or loadFromMap() after it has been populated.

In mapNodeModules(), additionalModuleLocations causes graphPackages() to be called against each location while reusing the Graph. The internal API has been changed somewhat to allow this. Because we do not know whether or we will be executing using the resulting CompartmentMapDescriptor (e.g. we can provide it to captureFromMap() which will not invoke dynamic requires), mapNodeModules() sets the CompartmentDescriptor.retained flag. When it does, it also sets the retained flag for each ModuleDescriptor found within CompartmentDescriptor.modules. It does this as the last step before returning the CompartmentMapDescriptor and only does so on CompartmentDescriptors not already found prior to graphing additional modules. This is a recursive operation which bails out of if either a) the CompartmentDescriptor was graphed by way of the entry module, or b) we've seen the CompartmentDescriptor before (to protect against cycles).

The supporting implementation in other modules is minimal excepting a change to loadFromMap where all additional packages are eagerly loaded via compartment.load(additionalPackageLocation).

Tests are concentrated in a new test suite (additional-modules.test.js) which covers the aforementioned APIs using the same comprehensive fixture.

Also:

• Fixed signature of chooseModuleDescriptor
• Fixed deprecation notice for compartmentMapForNodeModules; it was previously on the options object, but it should only be on the export.

Questions for Reviewers

• Naming is hard and I was not sure what to name the new options. The AdditionalPackageDetails type looks like the existing PackageDetails type (with a new field) thus influencing the naming—but they are otherwise unrelated and because of this I did not extend PackageDetails.

• I am monkeying with imports in a RecordModuleDescriptor object in order to direct SES' module loading. Is there a better way? The things I stuff in imports are also extremely dubious. I am not doing this anymore. Do not worry about it.

• Because additional packages are crawled similarly to the entry package and we do not tell Endo what package should be expected to import the additional package, the CompartmentDescriptor.path for this package would naturally be [] w/o any other intervention.

  Since that seems obviously wrong to me—because only the entry CompartmentDescriptor can have an empty path—it is now always set to [CompartmentDescriptor.name].

  IMO this is less wrong; the additional package must be reachable (at runtime) via the entry package (otherwise why would we specify it as an additional package?), but the additional package is not necessarily directly reachable (i.e. not a direct dependency) from the entry package.

  This has an impact on shortest path calculation, but I feel like a stable value for CompartmentDescriptor.path is more important than a logically-consistent one.

• Given the previous point, consider: the only way we would know which package directly (dynamically) imports this "additional" package is if the user explicitly told us which. One way to do this is via package policy, but we do not currently consider policy to be an indicator of a dependency relationship. Further, policy is not required to leverage additional packages.

  If we cannot use policy (though it would seem handy!), we would need to do something like change the shape of AdditionalModuleLocationObject (see above) to allow a new field which would presumably be the package location of whatever directly imports the module provided in AdditionalModuleLocationObject.location. If we had that information, we could use it to build logically consistent paths.

• @naugtur's implementation was using a special prefix in the canonical name for these "additional packages". I don't know what purpose this serves (other than he mentioned something about LavaMoat having some historical support for it).

Tangentially: since we expect policy to contain canonical names which have not yet been computed (which is backwards IMO, but I understand the tradeoffs), we should catch unused and/or unexpectedly empty package policies and report them. @lavamoat/node does some of this work already, but I feel like it should be happening in @endo/compartment-mapper instead.

Security Considerations

n/a

These changes do not meddle with systems loading untrusted code, afaik.

Scaling Considerations

Use of the new option is not free.

Documentation Considerations

n/a

Testing Considerations

n/a

Compatibility Considerations

No.

Upgrade Considerations

This warrants an entry in NEWS.md since it is part of a public API.

[boneskull]

the only way for this to happen would be to provide a broken CompartmentMapDescriptor in the first place. IDK if it's worth worrying about.

[naugtur]

Review in progress, but I think there might be a less (opinion) invasive way to serve the same need.

That being said, I want to encourage others to review because this exists and my hypothetical other way doesn't (at least until I try)

[boneskull]

This is gonna look like a mashup of #2864 and #2878. I am actually going to change this PR to target #2878.

[boneskull]

@naugtur Regarding setting the retained flag:

AFAICT it is needed, since we do not know what will happen with the compartment descriptor map after it exits mapNodeModules().

If it is provided to loadFromMap(), then you're right; setting it shouldn't be necessary.

If it is provided to captureFromMap(), then it is necessary because no execution occurs and there's no other way to tell the "digest" function that we should keep these compartments around.

Whether or not to set this bit could potentially be a flag to mapNodeModules, I suppose.

The latest change will actually set this flag recursively. It should be possible to narrow the set by determining which compartment descriptors are only referenced by the additional modules and other like descriptors. I haven't done that yet, but I think it makes sense to do since the additional module may reference the entry point (so we've just flagged the whole compartment map).

An alternative design (which may be a better idea, but idk) would involve fiddling with the digest function to force it to keep certain compartments.

[boneskull]

Also: I noticed that we need explicit control over the dev (and possibly conditions, though I don't use it personally) flag for each additional module.

For example, if our entrypoint is node_modules/.bin/webpack, we unequivocally do not want to map its dev deps. However the package containing our config file ./webpack.config.js should have its dev deps looked at!

[boneskull]

working on sorting out the merge conflict

[boneskull]

I'm going to close this in lieu of a forthcoming PR.