ng build and ng serve use inconsistent methods to determine files emitted from webpack

[spencerb02]

🐞 Bug report

Command (mark with an x)

• new
• build
• serve
• test
• e2e
• generate
• add
• update
• lint
• xi18n
• run
• config
• help
• version
• doc

Is this a regression?

Yes, the previous version in which this bug was not present was:

@angular-devkit/build-angular: "0.800.6"
@angular/cli: "8.0.6"

I apologize I didn't do the leg work to discover exactly which version it broke in. I was using v8.0.6 and was updating to latest v8 on my way to v9 (soon).

Description

A clear and concise description of the problem...

In the latest version of ng 8 the emitted files from the webpack build are handled differently by ng build than by ng serve. This was not the case in 8.0.6. (I have not tested this in ng 9 nor have I tested ng test).

This is due to this code being removed from @angular-devkit/build-webpack/src/utils.js:

// entrypoints might have multiple outputs
// such as runtime.js
for (const [name, entrypoint] of compilation.entrypoints) {
    const entryFiles = (entrypoint && entrypoint.getFiles()) || [];
    for (const file of entryFiles) {
        files.push({ name, file, extension: path.extname(file), initial: true });
    }
}

commit: e2b1905#diff-4e79ab2e1e76b5c5a42a38b4f3b50036

The getEmittedFiles method is how ng build determines the emitted files. In 8.0.6 it used to traverse the compilations entry points and then the chunks. Now in 8.3.25 it just traverses the chunks. ng serve, however, still traverses the endpoints:
@angular-devkit/build-angular/src/angular-cli-files/plugins/index-html-webpack-plugin.js

for (const [entryName, entrypoint] of compilation.entrypoints) {
  const entryFiles = ((entrypoint && entrypoint.getFiles()) || []).map((f) => ({
    name: entryName,
    file: f,
    extension: path.extname(f),
  }));
  //...
}

The outcome of this issue is that additional outputs of the main entrypoint are no longer added to the index.html when running ng build but they are included when I run ng serve which created false positives. (and lead to some really fun "works on my box" discussions with the quality engineer)

🔬 Minimal Reproduction

This is where it gets messy and I admit that I'm using a build extension. I'm adding to the webpack config through an ngx-build-plus plugin. Utilizing the plugin I add the following merge to the webpack config:

const merge = require("webpack-merge");

exports.default = {
  pre(options) {
    // ...
  },
  config(config) {
    const mergeStrategy = merge.strategy({
      // ...,
      "config.optimization.splitChunks.cacheGroups": "append",
    });

    return mergeStrategy(config, {
      optimization: {
        splitChunks: {
          cacheGroups: {
            angularJS: {
              name: "angularJS",
              chunks: "initial",
              test: /[\\/]node_modules[\\/](angular-?|ng-)/,
              priority: 5,
            },
            angular: {
              name: "angular",
              chunks: "initial",
              test: /[\\/]node_modules[\\/](@angular)[\\/]/,
              priority: 5,
            },
          },
        },
      },
    });
  },
  post(options) {
    // ... 
  },
};

This creates 2 new bundles angular.[hash:20].js and angularJS.[hash:20].js (did I mention it's a hybrid app?) which are initial bundles which are output by the main entrypoint. When I run ng serve they are included in the index.html because they are attached to the main endpoint. When I run ng build, they are not included in the index.html.

With ngx-build-plus installed and assuming the above plugin is in a file called config/webpack-loaders/ajs-plugin.js run the build:
ng build --plugin ~config/webpack-loaders/ajs-plugin.js or
ng serve --plugin ~config/webpack-loaders/ajs-plugin.js -o

I know that what I need to do is use the post step to enforce the cachegroups get added the way I expect, but I wanted to call out the inconsistency between the commands.

🌍 Your Environment

// OLD VERSION
Angular CLI: 8.0.6
Node: 12.16.1
OS: win32 x64
Angular: 8.2.14
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router, upgrade

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.800.6
@angular-devkit/build-angular     0.800.6
@angular-devkit/build-optimizer   0.800.6
@angular-devkit/build-webpack     0.800.6
@angular-devkit/core              8.0.0
@angular-devkit/schematics        8.0.0
@angular/cli                      8.0.6
@ngtools/webpack                  8.0.6
@schematics/angular               8.0.0
@schematics/update                0.800.6
rxjs                              6.4.0
typescript                        3.4.5
webpack                           4.30.0


// NEW VERSION
Angular CLI: 8.3.25
Node: 12.16.1
OS: win32 x64
Angular: 8.2.14
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router, upgrade

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.803.25
@angular-devkit/build-angular     0.803.25
@angular-devkit/build-optimizer   0.803.25
@angular-devkit/build-webpack     0.803.25
@angular-devkit/core              8.0.0
@angular-devkit/schematics        8.0.0
@angular/cli                      8.3.25
@ngtools/webpack                  8.3.25
@schematics/angular               8.0.0
@schematics/update                0.803.25
rxjs                              6.4.0
typescript                        3.5.3
webpack                           4.39.2

Angular json build options:

"architect": {
  "build": {
    "builder": "ngx-build-plus:browser",
    "options": {
      "outputPath": "dist/website",
      "index": "config/templates/index.html",
      "main": "src/main.ts",
      "polyfills": "src/polyfills.ts",
      "tsConfig": "tsconfig.app.json",
      "aot": false,
      "showCircularDependencies": false,
      "assets": [
        "src/assets",
      ],
      "styles": [
        "src/styles.scss"
      ],
    },
  },
  "configurations": {
    "production": {
      "fileReplacements": [{
        "replace": "src/environments/environment.ts",
        "with": "src/environments/environment.prod.ts"
      }],
      "optimization": true,
      "outputHashing": "all",
      "sourceMap": false,
      "extractCss": true,
      "namedChunks": false,
      "aot": true,
      "extractLicenses": true,
      "vendorChunk": true,
      "forkTypeChecker": true,
      "buildOptimizer": true,
      "budgets": []
    },
  },
  "serve": {
    "builder": "ngx-build-plus:dev-server",
    "options": {
      "browserTarget": "website:build"
    },
    "configurations": {
      "production": {
        "browserTarget": "website:build:production"
      },
      "staging": {
        "browserTarget": "website:build:staging"
      }
    }
  },
}

Anything else relevant?

All paths lead through @angular-devkit\build-angular\src\angular-cli-files\utilities\index-file\augment-index-html.js augmentIndexHtml() and the array of files that gets passed to it.

When running ng serve files is set to an array like this:

[
  { name: 'main', file: 'runtime.js', extension: '.js' },
  { name: 'main', file: 'angularJS.js', extension: '.js' },
  { name: 'main', file: 'angular.js', extension: '.js' },
  { name: 'main', file: 'vendor.js', extension: '.js' },
  { name: 'main', file: 'main.js', extension: '.js' },
  { name: 'polyfills', file: 'runtime.js', extension: '.js' },
  { name: 'polyfills', file: 'polyfills.js', extension: '.js' },
  { name: 'styles', file: 'runtime.js', extension: '.js' },
  { name: 'styles', file: 'styles.js', extension: '.js' },
]

Notice Angular and AngularJS have the name of main. This is because the name is set as the name of the entrypoint the emittedFile was found in.

When running ng build the same files are built (according to webpack-bundle-analyzer) but the files array looks like this:

[
  { file: 'runtime.bc9d7222a5be40bb49f2.js', extension: '.js', name: 'runtime' },
  { file: 'angularJS.bd8c6b36a4d831de98ad.js', extension: '.js', name: 'angularJS~main' },
  { file: 'angular.2378ac52c8e97a70ffb9.js', extension: '.js', name: 'angular~main' },
  { file: 'main.776ae6e317ff60328da0.js', extension: '.js', name: 'main' },
  { file: 'polyfills.83265812522305ef22ce.js', extension: '.js', name: 'polyfills' },
  { file: 'styles.2cdb9ded7df542f838e1.css', extension: '.css', name: 'styles' },
  { file: 'vendor.7dc8e0ec3a85cc2dc3fc.js', extension: '.js', name: 'vendor' }
]

Note that the name of these emittedFiles are the chunk name and not the entrypoint. These arrays of emittedFiles are then compared to the array of entrypoints to determine which ones are to be added to index.html:

[
  'polyfills-nomodule-es5',
  'polyfills-es5',
  'polyfills',
  'styles',
  'vendor',
  'main'
]

So ng serve recognizes the angular bundles are part of the main entrypoint and adds them to
index.html accordingly. ng build on the other hand doesn't because the chunk names are not the same as an entrypoint in the hardcoded list of entrypoints.

In conclusion:

1. I need to make my use of the build extension more robust and ensure the cachegroup bundles are added to the index.html in the plugin's post step.
2. I don't think ng serve and ng build should be different in how they map the emitted files.
3. I am very open to hearing the "right way" of doing things if there is a better/more accepted approach for handling large, eagerly loaded 3rd party bundles.

[aaronfrost]

I am guessing that they will need a reproducible scenario in order to fix this issue. It likely broke a very small number of people (given that all tests at Google passed). But... if it is still broken in 9... then they would want to fix it. I would recommend a small test project to reproduce in order for the team to look at it.

[clydin]

This cannot affect the functionality of the CLI when used directly with officially supported build configurations. Generating the described behavior is only possible when using the experimental extension API. The internal webpack configuration is specifically tailored to function with the CLI build system. Modifying it is not officially supported and can result in errant behavior.

With that said, the index HTML generation code is in the process of being refactored and the sorting code mentioned will no longer also act as a filter in the updated version. This should solve the issue as described. However, a timeframe for inclusion of the refactor has not yet been determined.

For point 3, can you provide some background on the Angular libraries being separated and eagerly loaded? With a newly generated 9.0 application, separating the Angular libraries with the configuration above results in an additional ~16.5kb of data that needs to be initially loaded. This is with a minimal set of Angular libraries and the difference should increase with additional libraries.

[spencerb02]

@clydin @alan-agius4 Thank you for your prompt response! I really appreciate how well Angular contributors and members take care of the community.

I'll admit that I don't understand all the nuances that go into the CLI build and webpack's part in it. Unfortunately, the experimental extension API that we are using can't be avoided in building our enterprise hybrid app that uses ngx-build-plus to bring both the Angular and AngularJS builds together in our ngUpgrade app.

I did note, however, in the commit where the snippet of code that was removed was originally added (55863a5#diff-4e79ab2e1e76b5c5a42a38b4f3b50036) the commit comment states:

Entrypoints might have other files associate with them such as runtime.js, it is is (sic) paramount to keep the relation between them especially when this result is needed to generate an index file

That seems to be the situation I am up against.

I am curious to know more about how an Angular 9 build is adversely affected by cache groups. The intent of the cache groups is they break up the rather large vendors (1.5+MB) bundle that is a result of all eager node_modules being added to the one bundle. This allows us to:

1. Utilize the benefits of http/2 and parallelize the transmission of the eagerly loaded bundles
2. Isolate them from the other bundles for caching and cache-busting purposes which makes a user's repeat visit only affected if we change the Angular(JS) version.

Ultimately, we are working towards a solution that will allow us to lazy load AngularJS and allow webpack to split it out naturally, but that is further down the road than Angular v9 for us, I'm sure.

[spencerb02]

Here is the repo to recreate the issue. I hope I have done this correctly. It is my first time.

https://github.com/spencerb02/angularCli-Issue17146

[KonanMentor]

But... if it is still broken in 9... then they would want to fix it.

I can confirm that the issue still exists in ng 9.

My use case involves @angular-builders/custom-webpack:browser builder though

[rolandschuster]

After the update from Angular 8 to Angular 9 I also have the described problem. No matter which package is used to extend the webpack configuration (@angular-builders/custom-webpack or ngx-build-plus). The resulting chunks are built by webpack but not linked in the index.html when using ng build. With ng serve everything works fine.

For licensing reasons we have to split the code into several files. We use Kendo UI, which is under a commercial license, and we don't want to have their code in one file with the code of all other third-party packages when we ship our product.

[spencerb02]

@clydin @alan-agius4 any update on this? I would really like to follow the recommendation from @StephenFluin at ng-conf to be on the latest Angular but this issue prevents that. Removing the cache groups is not a viable option.

So what are our options? Do we need to override the index.html creation to use HtmlWebpackPlugin so all chunks get included? I might be in favor of this approach because I have some other plugins I wouldn't mind using...

[rolandschuster]

So what are our options? Do we need to override the index.html creation to use HtmlWebpackPlugin so all chunks get included? I might be in favor of this approach because I have some other plugins I wouldn't mind using...

I also think that it should still be possible to define custom cache groups and that they should also be properly linked in the index.html, and I hope the Angular team is somehow working in that direction.

Until the issue is solved, I see a quick fix that can serve as a temporary solution to update to Angular 9 and have all cacheGroups still linked in the index.html: You could add an entry in the angular.json under "scripts" that has the name of the custom cacheGroup and link it to an empty file.

"scripts": [
  {
    "input": "src/emptyScript.js",
    "bundleName": "cacheGroupName"
  }
]

The appropriate cacheGroup in your webpack-config would look like this

cacheGroups: {
  cacheGroupName: {
    name: 'cacheGroupName',
    chunks: 'initial',
    ...
  }
}

This causes the link to the file 'cacheGroupName.js' in the index.html to be added. This is of course not a nice solution, but it will work temporarily.