AoT hangs forever after updating from v6 to v7 (or v8)

[runeabrahams1]

🐞 Bug report

Command (mark with an x)

- [ ] new
- [x] build
- [x] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Is this a regression?

Yes, the previous version in which this bug was not present was: v6 (latest)

Description

We got a quite large hybrid app (700+ Components, ~100 angularJs components/directives).
Build it with AoT works perfectly in Angular v6, but stops working in v7 or v8.
Building it without AoT works in all versions without issues.

The problem is that we get no errors when building with AoT in v7 or v8. The build get stuck at 11% and will stay there even after 50+ hours.

This is the line it get stuck on:
11% building 13/14 modules 1 active ...epos\Stream\node_modules\less-loader\dist\cjs.js??ref--14-3!C:\Repos\Stream\src\styles.less

Is there any way to get a more detailed log of what the compiler is trying to do?

I'm out of ideas of how to fix this. I've tried looking at other issues with AoT or UpgradeModule, but none of them quite matches this or helps with my issue.

I've done everything in https://update.angular.io/#6.1:7.0.
We run RxJS 6 without compat, and have removed HttpModule from everything

🔬 Minimal Reproduction

This is the problem. I can't reproduce it in a new app with all our third party libs and styles included. Something in out code breaks AoT compilation in version 7 and 8.

We run our code with this line:
node --max_old_space_size=16386 ./node_modules/@angular/cli/bin/ng serve --port 3000

🌍 Your Environment

I tried v7@latest, v8, v8.1, v8@next

Angular Version:

Angular CLI: 8.2.0-next.0
Node: 10.15.3
OS: win32 x64
Angular: 8.2.0-next.1
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router, upgrade

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.802.0-next.0
@angular-devkit/build-angular     0.802.0-next.0
@angular-devkit/build-optimizer   0.802.0-next.0
@angular-devkit/build-webpack     0.802.0-next.0
@angular-devkit/core              8.2.0-next.0
@angular-devkit/schematics        8.2.0-next.0
@angular/cli                      8.2.0-next.0
@ngtools/webpack                  8.2.0-next.0
@schematics/angular               8.2.0-next.0
@schematics/update                0.802.0-next.0
rxjs                              6.5.2
typescript                        3.4.5
webpack                           4.35.3

"dependencies": {
    "@angular/animations": "8.2.0-next.1",
    "@angular/common": "8.2.0-next.1",
    "@angular/compiler": "8.2.0-next.1",
    "@angular/core": "8.2.0-next.1",
    "@angular/forms": "8.2.0-next.1",
    "@angular/platform-browser": "8.2.0-next.1",
    "@angular/platform-browser-dynamic": "8.2.0-next.1",
    "@angular/router": "8.2.0-next.1",
    "@angular/upgrade": "8.2.0-next.1",
    "@swimlane/ngx-datatable": "~15.0.2",
    "@yellowspot/ng-truncate": "^1.5.0",
    "angular": "~1.7.8",
    "angular-chart.js": "1.1.1",
    "angular-dynamic-locale": "~0.1.37",
    "angular-elastic": "~2.5.1",
    "angular-file-upload": "2.5.0",
    "angular-filter": "~0.5.17",
    "angular-i18n": "~1.7.8",
    "angular-l10n": "8.0.0",
    "angular-notification": "1.1.1",
    "angular-sanitize": "~1.7.8",
    "angular-smart-table": "2.1.11",
    "angular-ui-bootstrap": "~2.5.6",
    "angular2-cookie-law": "7.0.1",
    "awesome-bootstrap-checkbox": "~0.3.7",
    "blueimp-gallery": "~2.33.0",
    "bootstrap": "^3.4.1",
    "chart.js": "~2.8.0",
    "clipboard": "2.0.4",
    "core-js": "^3.1.4",
    "devextreme": "19.1.4",
    "devextreme-angular": "19.1.4",
    "devextreme-intl": "19.1.4",
    "dragula": "3.7.2",
    "dragular": "1.3.2",
    "file-saver": "2.0.2",
    "font-awesome": "4.7.0",
    "fraction.js": "~4.0.12",
    "hammerjs": "^2.0.8",
    "jquery": "^3.4.1",
    "moment": "~2.24.0",
    "msal": "1.0.2",
    "ng-tags-input": "3.2.0",
    "ng2-charts": "^2.3.0",
    "ng2-dragula": "2.1.1",
    "ng2-file-upload": "1.3.0",
    "ng2-select": "2.0.0",
    "nglocale": "^1.0.0",
    "ngx-bootstrap": "^5.1.0",
    "ngx-clipboard": "12.2.0",
    "ngx-cookie": "~4.0.2",
    "ngx-icon-picker": "^0.0.10",
    "ngx-quill": "~7.1.0",
    "pace-progress": "~1.0.2",
    "powerbi-client": "^2.7.5",
    "quill": "^1.3.6",
    "rxjs": "~6.5.2",
    "signalr": "~2.4.1",
    "simplebar": "<4",
    "tslib": "^1.9.0",
    "underscore": "1.9.1",
    "zone.js": "~0.9.1"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.802.0-next.0",
    "@angular/cli": "8.2.0-next.0",
    "@angular/compiler-cli": "8.2.0-next.1",
    "@angular/language-service": "8.2.0-next.1",
    "@babel/core": "^7.5.0",
    "@babel/preset-env": "^7.5.0",
    "@babel/register": "^7.4.4",
    "@cypress/webpack-preprocessor": "4.1.0",
    "@types/angular-file-upload": "^2.5.1",
    "@types/angular-signalr-hub": "^1.5.32",
    "@types/angular-ui-bootstrap": "^0.13.43",
    "@types/chai": "^4.1.2",
    "@types/file-saver": "^2.0.1",
    "@types/google.analytics": "^0.0.39",
    "@types/jasmine": "~3.3.0",
    "@types/jasminewd2": "~2.0.3",
    "@types/jquery": "^3.2.9",
    "@types/mocha": "^5.0.0",
    "@types/ng-file-upload": "^12.2.1",
    "@types/node": "^10.9.4",
    "@types/signalr": "^2.2.34",
    "@types/underscore": "^1.9.0",
    "chai": "^4.2.0",
    "codelyzer": "^5.0.1",
    "concurrently": "^4.1.1",
    "cypress": "~3.4.0",
    "fs-extra": "~8.1.0",
    "gulp": "^4.0.0",
    "gulp-angular-templatecache": "~3.0.0",
    "gulp-minify-html": "1.0.6",
    "husky": "^3.0.0",
    "jasmine-core": "~3.4.0",
    "jasmine-spec-reporter": "~4.2.1",
    "karma": "~4.2.0",
    "karma-chrome-launcher": "~3.0.0",
    "karma-coverage-istanbul-reporter": "~2.0.5",
    "karma-jasmine": "~2.0.1",
    "karma-jasmine-html-reporter": "^1.4.2",
    "lint-staged": "~9.2.0",
    "prettier": "~1.18.2",
    "prettier-tslint": "~0.4.2",
    "recursive-readdir": "2.2.2",
    "rxjs-tslint-rules": "~4.24.3",
    "ts-loader": "^6.0.4",
    "ts-node": "~8.3.0",
    "tslint": "~5.18.0",
    "tslint-config-prettier": "^1.18.0",
    "typescript": "3.4.5",
    "typescript-tslint-plugin": "~0.5.2",
    "webpack": "^4.35.0",
    "webpack-bundle-analyzer": "^3.3.2",
    "yarn": "~1.17.0"
  }

[szelag-michal]

I have this same issue. Stuck on 19% styles.scss

{
  "name": "pegasus-ui-admin",
  "version": "0.0.0",
  "license": "MIT",
  "scripts": {
    "ng": "node node_modules/.bin/ng",
    "serve-test": "node node_modules/.bin/ng serve --proxy-config proxy.conf.test.json --open true",
    "serve-qa": "node node_modules/.bin/ng serve --proxy-config proxy.conf.qa.json --open true",
    "serve-local": "node node_modules/.bin/ng serve --proxy-config proxy.conf.local.json --open true",
    "serve-qa2": "node node_modules/.bin/ng serve --proxy-config proxy.conf.qa2.json --open true",
    "serve-ui": "node node_modules/.bin/ng serve --proxy-config proxy.conf.ui.json --open true",
    "serve-bodemo": "node node_modules/.bin/ng serve --proxy-config proxy.conf.bodemo.json --open true",
    "start": "node node_modules/.bin/ng serve --proxy-config proxy.conf.json --open true",
    "build": "node node_modules/.bin/ng build",
    "test": "node node_modules/.bin/ng test",
    "test-prod": "node node_modules/.bin/ng test --single-run",
    "lint": "node node_modules/.bin/ng lint",
    "e2e": "node node_modules/.bin/ng e2e",
    "prod": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build --prod",
    "proper-prod": "node node_modules/.bin/ng build --prod && node node_modules/.bin/ng test --single-run"
  },
  "private": true,
  "dependencies": {
    "@angular-devkit/build-angular": "0.801.2",
    "@angular/animations": "8.1.2",
    "@angular/cdk": "8.1.1",
    "@angular/common": "8.1.2",
    "@angular/compiler": "=8.1.0",
    "@angular/core": "8.1.2",
    "@angular/forms": "8.1.2",
    "@angular/http": "=7.2.15",
    "@angular/material": "8.1.1",
    "@angular/platform-browser": "8.1.2",
    "@angular/platform-browser-dynamic": "8.1.2",
    "@angular/router": "8.1.2",
    "@ngrx/effects": "8.1.0",
    "@ngrx/router-store": "8.1.0",
    "@ngrx/store": "8.1.0",
    "@ngrx/store-devtools": "8.1.0",
    "angular-ui-notification": "=0.3.6",
    "angular2-text-mask": "=9.0.0",
    "angular2-uuid": "=1.1.1",
    "bootstrap": "4.3.1",
    "core-js": "3.1.4",
    "font-awesome": "=4.7.0",
    "jquery": "=3.4.1",
    "lodash": "4.17.14",
    "moment": "=2.24.0",
    "ng2-slim-loading-bar": "=4.0.0",
    "ng2-toasty": "=4.0.3",
    "ngx-bootstrap": "5.1.0",
    "node-sass": "=4.12.0",
    "npm": "6.10.1",
    "path": "=0.12.7",
    "rxjs": "=6.5.2",
    "rxjs-compat": "=6.5.2",
    "select2": "=4.0.7",
    "ui-select": "=0.19.8",
    "zone.js": "=0.9.1"
  },
  "devDependencies": {
    "@angular/cli": "^8.1.2",
    "@angular/compiler-cli": "8.1.2",
    "@angular/language-service": "8.1.2",
    "@types/jasmine": "=3.3.13",
    "@types/jasminewd2": "=2.0.6",
    "@types/node": "12.6.8",
    "codelyzer": "=5.1.0",
    "jasmine-core": "=3.4.0",
    "jasmine-spec-reporter": "=4.2.1",
    "karma": "4.2.0",
    "karma-chrome-launcher": "3.0.0",
    "karma-cli": "=2.0.0",
    "karma-coverage-istanbul-reporter": "=2.0.5",
    "karma-jasmine": "=2.0.1",
    "karma-jasmine-html-reporter": "=1.4.2",
    "node-less": "=1.0.0",
    "protractor": "=5.4.2",
    "ts-node": "=8.3.0",
    "tslint": "=5.18.0",
    "typescript": "3.4.5"
  }
}

[filipesilva]

Heya, I don't really know what's happening in your cases. If I had to bet, maybe it's some TS or Sass/Less bug.

If you can't provide me with a repro, I can tell you how to profile your app and maybe you can see what's happening. You can use https://gist.github.com/filipesilva/9ff23fb136af2a192f42d3a2498dc6ef to profile your apps.

The CPU profile is your best bet. Since your process never really ends you'll need to manually add a timeout too, something like this:

      // Save the CPU profile after execution has finished.
      after = writeCPUProfile;
      setTimeout(async () => {
        await writeCPUProfile();
        process.exit(1);
      }, 3 * 60 * 1000);

[runeabrahams1]

@filipesilva I tried you code.
If I set time timeout to before it get stuck, but after compiler start I get a log file and it stops due to the exit() all after.
But if the code gets to 11%, then the profile is never saved and its stuck forever.

Now the PC is using all my memory too, I guess the function-call at 11% never finishes or something.

You got some ideas on what to do now?

[filipesilva]

If whatever is happening never finishes and is also synchronous, then the setTimeout code will never run.

Another alternative is to run node --inspect-brk --max_old_space_size=16386 ./node_modules/@angular/cli/bin/ng serve --port 3000 (note the added --inspect-brk). Then you can open the chrome CPU profiler and start/stop profiling manually. You will need to go into the "Source" tab and resume code execution to start.

[runeabrahams1]

CPU-20190722T102036.zip

@filipesilva Here is a CPU profile that ran for around 3-5 secs. I'm not sure what the problem is, but it seems to me that typescript is the issue.

The ForEach loop seems to continue forever as I can attach the profiler again and its still in that loop.

[runeabrahams1]

This is a stacktrace of where it gets stuck. Angular async awaits on the _emit function forever

[filipesilva]

I don't know if it's related but TS 3.4 had a problem with union types which caused massively degraded performance in some cases (microsoft/TypeScript#31584).

Can you try with TS 3.5? If you get an error about unsupported TS versions you can use this toplevel TS property to bypass it: https://github.com/angular/angular/blob/cb848b94102855d1e773892b25375304102cab69/aio/tsconfig.app.json#L16-L18

[runeabrahams1]

That fixed the problem.
I guess we can leave the flag on and use TS 3.5 for now until Angular adds support for 3.5.

[filipesilva]

Glad to hear it! We're updating the framework in angular/angular#31615, so it should be supported before version 9.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}