Bundling pre-ES6 modules

[callumlocke]

Hi Rich. This project looks very exciting. I'm sold.

I have a question about pre-ES6 modules... I tried importing lodash as an experiment (today's lodash), and got this error:

Package lodash does not have a jsnext:main field, and so cannot be included in your rollup. Try adding it as an external module instead (e.g. options.external = ['lodash']

Is this a permanent design decision or is it just early days? Will you ever add functionality that can read an old-style main script and traverse require() calls...or do you recommend another strategy for people who want to mix ES6 and old school modules in one bundle (e.g. using Browserify as another build step after rollup)?

[Rich-Harris]

I've gone back and forth on this. For the time being I think it's better to focus solely on ES6 modules, at least until we iron out all the kinks, but long term we definitely need a sane way to handle legacy module formats.

In theory, it can be done by overriding the resolveId function to accept CommonJS modules, and adding a transform step that wraps or rewrites modules so that Rollup can analyse them. To what extent you can convert CommonJS etc, I don't know - in a lot of cases it'd be straightforward, but I've no idea how you'd handle things like this (it's not quite as contrived as it looks)...

var myLib = exports;
myLib.foo = 'foo';

augment(myLib);
function augment(lib) { lib.bar = 'bar'; }

...so a strategy of wrapping rather than rewriting is probably better.

Luckily @guybedford has spent a lot of time thinking about these problems, and has done some initial research on incorporating Rollup's strategy inside JSPM: systemjs/builder#199. This seems like the best route forward, since JSPM is a soup-to-nuts solution (repository, bundler, module loader polyfill etc all talking the same language) that already handles legacy formats.

For right now though, my workaround is to have a two-phase bundling process, exactly as you suggest. First, I bundle my app code (plus any npm depedencies that expose jsnext:main, i.e. anything I wrote myself 😆) as CommonJS, and then I pass that bundle over to Browserify (but could be Webpack or whatever), which takes it from there. It works perfectly well, it's just a PITA to have to juggle all these different tools which is why I'm excited to see where JSPM goes.

[Victorystick]

Actually, @callumlocke, if you want an to import lodash into your ES6 project, you can just include the ES6 build of lodash. On npm it's called lodash-es. It has a "jsnext:main" field defined, so it should be ready to go. I've been able import it in my bundles, and once #47 is approved, the only hiccup I've encountered with it so far should be resolved.

[ha404]

I'm pretty confused, does this replace Webpack? Thanks!

[Victorystick]

The wiki says it better than I can.

[callumlocke]

@Rich-Harris:

First, I bundle my app code (plus any npm depedencies that expose jsnext:main, i.e. anything I wrote myself 😆) as CommonJS, and then I pass that bundle over to Browserify (but could be Webpack or whatever), which takes it from there.

That sounds good! Just to clarify, with this rollup-then-browserify approach, would your source code contain contain a mixture of import statements and require calls? ...i.e. use import for your own local ES6 modules AND for third party modules that expose a jsnext:main, but use require for everything else)?

[Rich-Harris]

@callumlocke you could, though more likely you'd write your app code all as ES6...

import external from 'third-party';
import foo from './my-utils';

/* ... *.

...and create a CommonJS bundle with Rollup, specifying which modules are external, then hand it over to Browserify:

rollup --input main.js\
       --format cjs\
       --external third-party,other-third-party\
       --output tmp/bundle.js

browserify tmp/bundle.js > dist/bundle.js

That's what I do, anyway!

[callumlocke]

Ahh ok, I did try that at first, but rollup errored out when it failed to resolve a third party import (which didn't have a jsnext:main), because I wasn't using the cjs format.

So basically the cjs format means: rollup will bundle whatever it can into the file, but for unresolvable imports it will just convert the import statement into a require call. Is that a good summary?

[Victorystick]

@callumlocke Yes, that's correct. What Rollup can't pull into the bundle will be "imported" using the idioms of the target module format.

// ES2015
import * as bar from './bar.js';

// CommonJS
var bar = require( './bar.js' );

// AMD
define(['./bar.js'], function (bar) {
  ...
});

// IIFE, use the `globals` option to tweak global names
(function (bar) {
  ...
)(bar);

And UMD being a combination of the last three.

[Hypercubed]

Hello @Rich-Harris (and others), Thank you for rollup. I'm using it in two of my new project.

Related to this issue. I'd really like to use rollup to pull in a few small helper utilities when building for the browser (for example a Object.assign shim and node.js util's isUndefined). Rollup seems ideal for this. Since rollup currently does not support importing from CJS modules what's the suggested temporary fix? The easy thing to do is copy and paste. We could also fork and convert a few core CJS modules to ES6 (https://github.com/defunctzombie/node-util/blob/master/util.js for example). Or wait for something else (e.g. future version of SystemJS builder using rollup).

I should add while I love JSPM for building apps I don't want to use JSPM (or Browserify) for building modules at this point. Rollup is much cleaner.

Edit: I also saw what you were doing with the *-jsnext repos.

[callumlocke]

...and create a CommonJS bundle with Rollup, specifying which modules are external, then hand it over to Browserify:

rollup --input main.js\
       --format cjs\
       --external third-party,other-third-party\
       --output tmp/bundle.js

browserify tmp/bundle.js > dist/bundle.js

Could there be a new option like --auto-externals, which would tell rollup to assume I want to treat any unresolvable imports as external?

[callumlocke]

@Victorystick:

Actually, @callumlocke, if you want an to import lodash into your ES6 project, you can just include the ES6 build of lodash. On npm it's called lodash-es. It has a "jsnext:main" field defined, so it should be ready to go. I've been able import it in my bundles, and once #47 is approved, the only hiccup I've encountered with it so far should be resolved.

Importing an individual method using e.g. import {zipObject} from 'lodash-es'; doesn't work (error: Module ~/foo/node_modules/lodash-es/lodash.js does not export zipObject).

Looks like this is because the module doesn't export zipObject from './array/zipObject'. Instead it just exports one big default (the entire lodash library). Is this going to be changed in lodash-es? Or is this a limitation of rollup, that it doesn't yet know how to grab a single property off a big default export in the absence of a named export?

[timshannon]

@callumlocke If you look, the main entry point for the lib just imports a bunch of smaller modules. What I've been doing (if I don't want to import the entire gigantic library) is just import the modules I want:

import findIndex from "../lib/lodash/array/findIndex";

It's been working great for me.

[Rich-Harris]

@Hypercubed

Since rollup currently does not support importing from CJS modules what's the suggested temporary fix?

Depending on how many helpers we're talking about, copy and paste might be the best option, if it's the difference between being able to keep the bundle entirely inside Rollup and having to pass it off to Browserify for a second step. Though I'm still wondering about the possibility of somehow transforming CommonJS modules into ES6 during the transformation phase.

Long-term though, the best option is to start creating ES6 modules and open sourcing them! 😀

@callumlocke

Could there be a new option like --auto-externals

Maybe... with Esperanto (Rollup's predecessor) I found that being permissive caused more problems than it solved (e.g. if you got the wrong number of ../s when importing ../../utils/foo, it would assume you were referencing an external module), though if it was an option perhaps that would be okay. I typically use the JS API rather than the CLI, and do this sort of thing:

{
  entry: 'app.js',
  format: 'cjs',
  external: (function () {
    var dependencies = require( './package.json' ).dependencies;

    return Object.keys( dependencies ).filter( function ( name ) {
      return !require( name + '/package.json' )[ 'jsnext:main' ];
    });
  }())
}

An option would be simpler.

Importing an individual method using e.g. import {zipObject} from 'lodash-es'; doesn't work

That's a shame. It would be great to be able to use lodash as a 'true' ES6 module and get the tree-shaking benefits without having to remember (or lookup) the fact that zipObject lives in the array folder. There would be some complicating factors, e.g. chaining would only work if you included the whole library (because everything needs to live on lodash.prototype), so it would actually be a different library. You'd have to ask @jdalton if he has any plans in that direction!

Or is this a limitation of rollup, that it doesn't yet know how to grab a single property off a big default export

It's more a limitation of static analysis in as dynamic a language as JavaScript, unfortunately.

@timshannon FYI in more recent versions of Rollup, if you have lodash installed as an npm module, you can do this if it's easier:

import findIndex from "lodash/array/findIndex";

[Victorystick]

@timshannon FYI in more recent versions of Rollup, if you have lodash installed as an npm module, you can do this if it's easier:

import findIndex from "lodash/array/findIndex";

This is how I've been doing it. Until lodash handles is exports properly.