Skip to content

[Experiment] Cache Dts Emit result / transform #49156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 17 commits into from
Closed

Conversation

sheetalkamat
Copy link
Member

Fixes #43995

@sheetalkamat
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 18, 2022

Heya @sheetalkamat, I've started to run the perf test suite on this PR at 5bba57e. You can monitor the build here.

Update: The results are in!

@typescript-bot typescript-bot added the For Milestone Bug PRs that fix a bug with a specific milestone label May 18, 2022
@sheetalkamat
Copy link
Member Author

Hmm Actually perf test is not useful since we dont do d.ts signatures any more as part of first run but only on changes.

@sheetalkamat
Copy link
Member Author

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 18, 2022

Heya @sheetalkamat, I've started to run the tarball bundle task on this PR at 5bba57e. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented May 18, 2022

Hey @sheetalkamat, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/126477/artifacts?artifactName=tgz&fileId=AB67468A4BDAFD4425DD4B3A623B9C658DA4D3C4BEE05A9788FA683B0835507802&fileName=/typescript-4.8.0-insiders.20220518.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

@typescript-bot
Copy link
Collaborator

@sheetalkamat
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..49156

Metric main 49156 Delta Best Worst
Angular - node (v10.16.3, x64)
Memory used 357,705k (± 0.03%) 357,705k (± 0.02%) +0k (+ 0.00%) 357,540k 357,897k
Parse Time 2.07s (± 0.32%) 2.07s (± 0.29%) -0.00s (- 0.05%) 2.05s 2.08s
Bind Time 0.86s (± 0.77%) 0.86s (± 0.78%) -0.01s (- 0.70%) 0.85s 0.88s
Check Time 5.83s (± 0.53%) 5.82s (± 0.64%) -0.01s (- 0.21%) 5.74s 5.89s
Emit Time 6.00s (± 0.21%) 6.00s (± 0.56%) -0.01s (- 0.08%) 5.91s 6.06s
Total Time 14.77s (± 0.27%) 14.74s (± 0.41%) -0.02s (- 0.16%) 14.65s 14.91s
Compiler-Unions - node (v10.16.3, x64)
Memory used 206,010k (± 0.02%) 206,008k (± 0.03%) -2k (- 0.00%) 205,883k 206,148k
Parse Time 0.84s (± 0.62%) 0.84s (± 0.74%) -0.00s (- 0.24%) 0.82s 0.85s
Bind Time 0.51s (± 1.02%) 0.51s (± 0.58%) +0.00s (+ 0.20%) 0.51s 0.52s
Check Time 7.92s (± 0.17%) 7.94s (± 0.66%) +0.02s (+ 0.19%) 7.84s 8.10s
Emit Time 2.50s (± 0.94%) 2.53s (± 0.80%) +0.03s (+ 1.04%) 2.48s 2.58s
Total Time 11.78s (± 0.26%) 11.82s (± 0.47%) +0.04s (+ 0.34%) 11.70s 12.00s
Monaco - node (v10.16.3, x64)
Memory used 343,841k (± 0.02%) 343,772k (± 0.02%) -69k (- 0.02%) 343,631k 343,904k
Parse Time 1.59s (± 0.58%) 1.59s (± 0.66%) -0.01s (- 0.31%) 1.56s 1.61s
Bind Time 0.75s (± 0.48%) 0.76s (± 0.44%) +0.01s (+ 0.80%) 0.75s 0.77s
Check Time 5.81s (± 0.42%) 5.81s (± 0.46%) -0.00s (- 0.09%) 5.76s 5.87s
Emit Time 3.25s (± 0.89%) 3.23s (± 0.99%) -0.02s (- 0.62%) 3.19s 3.34s
Total Time 11.41s (± 0.46%) 11.39s (± 0.34%) -0.03s (- 0.24%) 11.30s 11.48s
TFS - node (v10.16.3, x64)
Memory used 305,496k (± 0.02%) 305,543k (± 0.01%) +47k (+ 0.02%) 305,454k 305,613k
Parse Time 1.29s (± 0.63%) 1.29s (± 0.46%) -0.01s (- 0.54%) 1.28s 1.30s
Bind Time 0.72s (± 0.51%) 0.71s (± 0.67%) -0.00s (- 0.42%) 0.70s 0.72s
Check Time 5.24s (± 0.46%) 5.26s (± 0.48%) +0.01s (+ 0.27%) 5.21s 5.30s
Emit Time 3.42s (± 1.11%) 3.44s (± 1.34%) +0.02s (+ 0.56%) 3.34s 3.53s
Total Time 10.67s (± 0.55%) 10.69s (± 0.46%) +0.02s (+ 0.21%) 10.58s 10.81s
material-ui - node (v10.16.3, x64)
Memory used 468,492k (± 0.01%) 468,562k (± 0.01%) +70k (+ 0.01%) 468,467k 468,675k
Parse Time 1.83s (± 0.60%) 1.83s (± 0.41%) -0.01s (- 0.27%) 1.81s 1.84s
Bind Time 0.67s (± 1.49%) 0.68s (± 0.50%) +0.00s (+ 0.45%) 0.67s 0.68s
Check Time 14.29s (± 0.37%) 14.33s (± 0.58%) +0.04s (+ 0.28%) 14.14s 14.52s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 16.80s (± 0.32%) 16.84s (± 0.51%) +0.04s (+ 0.22%) 16.64s 17.03s
xstate - node (v10.16.3, x64)
Memory used 575,305k (± 1.31%) 571,914k (± 0.01%) -3,392k (- 0.59%) 571,725k 572,098k
Parse Time 2.58s (± 0.14%) 2.59s (± 0.27%) +0.00s (+ 0.19%) 2.58s 2.61s
Bind Time 1.01s (± 0.49%) 1.00s (± 0.37%) -0.00s (- 0.10%) 1.00s 1.01s
Check Time 1.53s (± 0.87%) 1.53s (± 0.52%) +0.00s (+ 0.07%) 1.51s 1.55s
Emit Time 0.07s (± 0.00%) 0.07s (± 3.14%) +0.00s (+ 1.43%) 0.07s 0.08s
Total Time 5.19s (± 0.36%) 5.19s (± 0.25%) +0.00s (+ 0.04%) 5.16s 5.21s
Angular - node (v12.1.0, x64)
Memory used 335,407k (± 0.02%) 335,362k (± 0.02%) -46k (- 0.01%) 335,264k 335,509k
Parse Time 2.07s (± 0.60%) 2.07s (± 0.42%) +0.00s (+ 0.00%) 2.05s 2.09s
Bind Time 0.82s (± 0.68%) 0.83s (± 0.88%) +0.01s (+ 0.73%) 0.82s 0.85s
Check Time 5.63s (± 0.37%) 5.64s (± 0.61%) +0.01s (+ 0.21%) 5.58s 5.72s
Emit Time 6.25s (± 0.74%) 6.23s (± 0.74%) -0.02s (- 0.37%) 6.14s 6.35s
Total Time 14.77s (± 0.35%) 14.76s (± 0.49%) -0.01s (- 0.05%) 14.59s 14.92s
Compiler-Unions - node (v12.1.0, x64)
Memory used 193,424k (± 0.15%) 193,558k (± 0.10%) +134k (+ 0.07%) 193,048k 193,817k
Parse Time 0.83s (± 1.09%) 0.83s (± 0.80%) +0.00s (+ 0.48%) 0.82s 0.85s
Bind Time 0.53s (± 0.94%) 0.53s (± 1.10%) +0.00s (+ 0.19%) 0.51s 0.54s
Check Time 7.45s (± 0.76%) 7.38s (± 0.35%) -0.07s (- 0.89%) 7.33s 7.44s
Emit Time 2.54s (± 0.78%) 2.53s (± 0.93%) -0.00s (- 0.08%) 2.49s 2.59s
Total Time 11.34s (± 0.65%) 11.28s (± 0.41%) -0.06s (- 0.55%) 11.19s 11.38s
Monaco - node (v12.1.0, x64)
Memory used 326,855k (± 0.02%) 326,735k (± 0.07%) -120k (- 0.04%) 325,831k 326,957k
Parse Time 1.56s (± 0.65%) 1.54s (± 0.72%) -0.02s (- 1.02%) 1.52s 1.57s
Bind Time 0.74s (± 0.65%) 0.74s (± 0.70%) +0.00s (+ 0.27%) 0.73s 0.75s
Check Time 5.63s (± 0.42%) 5.65s (± 0.59%) +0.02s (+ 0.30%) 5.60s 5.75s
Emit Time 3.27s (± 0.28%) 3.25s (± 0.71%) -0.02s (- 0.58%) 3.20s 3.32s
Total Time 11.20s (± 0.25%) 11.19s (± 0.43%) -0.01s (- 0.12%) 11.07s 11.32s
TFS - node (v12.1.0, x64)
Memory used 290,167k (± 0.02%) 290,194k (± 0.02%) +27k (+ 0.01%) 290,081k 290,344k
Parse Time 1.29s (± 0.69%) 1.29s (± 0.81%) -0.00s (- 0.15%) 1.28s 1.33s
Bind Time 0.72s (± 0.82%) 0.73s (± 1.19%) +0.00s (+ 0.55%) 0.71s 0.75s
Check Time 5.22s (± 0.49%) 5.20s (± 0.39%) -0.02s (- 0.44%) 5.14s 5.23s
Emit Time 3.50s (± 0.56%) 3.47s (± 0.69%) -0.02s (- 0.66%) 3.40s 3.51s
Total Time 10.74s (± 0.41%) 10.70s (± 0.26%) -0.04s (- 0.41%) 10.64s 10.77s
material-ui - node (v12.1.0, x64)
Memory used 447,550k (± 0.02%) 447,572k (± 0.01%) +22k (+ 0.00%) 447,423k 447,730k
Parse Time 1.83s (± 0.41%) 1.82s (± 0.42%) -0.00s (- 0.05%) 1.81s 1.84s
Bind Time 0.65s (± 0.80%) 0.65s (± 0.46%) +0.00s (+ 0.15%) 0.65s 0.66s
Check Time 12.88s (± 0.32%) 12.90s (± 0.36%) +0.02s (+ 0.16%) 12.82s 13.01s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.36s (± 0.28%) 15.38s (± 0.31%) +0.02s (+ 0.16%) 15.30s 15.50s
xstate - node (v12.1.0, x64)
Memory used 537,510k (± 0.02%) 537,557k (± 0.02%) +47k (+ 0.01%) 537,318k 537,798k
Parse Time 2.53s (± 0.36%) 2.54s (± 0.55%) +0.01s (+ 0.40%) 2.52s 2.58s
Bind Time 1.02s (± 1.04%) 1.03s (± 0.56%) +0.00s (+ 0.20%) 1.02s 1.04s
Check Time 1.47s (± 0.53%) 1.47s (± 0.85%) +0.00s (+ 0.14%) 1.46s 1.51s
Emit Time 0.07s (± 0.00%) 0.07s (± 0.00%) 0.00s ( 0.00%) 0.07s 0.07s
Total Time 5.10s (± 0.36%) 5.11s (± 0.35%) +0.01s (+ 0.29%) 5.09s 5.17s
Angular - node (v14.15.1, x64)
Memory used 333,635k (± 0.01%) 333,634k (± 0.01%) -2k (- 0.00%) 333,593k 333,689k
Parse Time 2.04s (± 0.39%) 2.04s (± 0.54%) +0.00s (+ 0.00%) 2.02s 2.07s
Bind Time 0.87s (± 1.01%) 0.87s (± 0.38%) -0.00s (- 0.34%) 0.86s 0.88s
Check Time 5.63s (± 0.57%) 5.62s (± 0.15%) -0.01s (- 0.14%) 5.61s 5.65s
Emit Time 6.30s (± 0.55%) 6.30s (± 0.64%) -0.00s (- 0.02%) 6.24s 6.40s
Total Time 14.84s (± 0.35%) 14.83s (± 0.25%) -0.01s (- 0.10%) 14.74s 14.91s
Compiler-Unions - node (v14.15.1, x64)
Memory used 192,268k (± 0.02%) 192,272k (± 0.02%) +4k (+ 0.00%) 192,181k 192,333k
Parse Time 0.85s (± 0.56%) 0.84s (± 0.43%) -0.00s (- 0.35%) 0.84s 0.85s
Bind Time 0.56s (± 0.59%) 0.56s (± 0.80%) -0.00s (- 0.36%) 0.55s 0.57s
Check Time 7.52s (± 0.39%) 7.51s (± 0.60%) -0.00s (- 0.04%) 7.44s 7.65s
Emit Time 2.51s (± 0.47%) 2.50s (± 1.20%) -0.01s (- 0.36%) 2.46s 2.60s
Total Time 11.43s (± 0.26%) 11.42s (± 0.51%) -0.01s (- 0.12%) 11.32s 11.59s
Monaco - node (v14.15.1, x64)
Memory used 325,634k (± 0.01%) 325,620k (± 0.00%) -14k (- 0.00%) 325,607k 325,650k
Parse Time 1.57s (± 0.84%) 1.57s (± 0.62%) +0.00s (+ 0.06%) 1.55s 1.59s
Bind Time 0.78s (± 1.34%) 0.77s (± 1.06%) -0.01s (- 0.64%) 0.76s 0.80s
Check Time 5.53s (± 0.29%) 5.53s (± 0.28%) +0.00s (+ 0.09%) 5.50s 5.57s
Emit Time 3.33s (± 0.71%) 3.32s (± 0.94%) -0.02s (- 0.54%) 3.27s 3.43s
Total Time 11.20s (± 0.35%) 11.19s (± 0.43%) -0.01s (- 0.12%) 11.11s 11.36s
TFS - node (v14.15.1, x64)
Memory used 289,138k (± 0.01%) 289,124k (± 0.01%) -14k (- 0.00%) 289,096k 289,158k
Parse Time 1.36s (± 1.73%) 1.36s (± 1.04%) -0.00s (- 0.07%) 1.34s 1.40s
Bind Time 0.72s (± 1.05%) 0.72s (± 1.03%) -0.00s (- 0.14%) 0.71s 0.75s
Check Time 5.22s (± 0.47%) 5.20s (± 0.48%) -0.02s (- 0.33%) 5.16s 5.27s
Emit Time 3.59s (± 1.20%) 3.51s (± 2.02%) -0.08s (- 2.23%) 3.37s 3.63s
Total Time 10.89s (± 0.46%) 10.79s (± 0.67%) -0.10s (- 0.91%) 10.66s 10.95s
material-ui - node (v14.15.1, x64)
Memory used 445,756k (± 0.01%) 445,617k (± 0.06%) -139k (- 0.03%) 444,479k 445,770k
Parse Time 1.86s (± 0.41%) 1.87s (± 0.41%) +0.01s (+ 0.32%) 1.86s 1.89s
Bind Time 0.70s (± 0.53%) 0.70s (± 0.82%) +0.00s (+ 0.72%) 0.69s 0.71s
Check Time 13.03s (± 0.71%) 13.02s (± 0.69%) -0.01s (- 0.06%) 12.86s 13.32s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.59s (± 0.60%) 15.59s (± 0.58%) -0.00s (- 0.01%) 15.42s 15.88s
xstate - node (v14.15.1, x64)
Memory used 535,264k (± 0.00%) 535,292k (± 0.01%) +28k (+ 0.01%) 535,208k 535,373k
Parse Time 2.58s (± 0.51%) 2.58s (± 0.56%) +0.00s (+ 0.12%) 2.56s 2.62s
Bind Time 1.15s (± 0.87%) 1.14s (± 0.39%) -0.01s (- 0.78%) 1.13s 1.15s
Check Time 1.52s (± 0.55%) 1.52s (± 0.22%) +0.00s (+ 0.07%) 1.51s 1.52s
Emit Time 0.07s (± 0.00%) 0.07s (± 0.00%) 0.00s ( 0.00%) 0.07s 0.07s
Total Time 5.32s (± 0.21%) 5.31s (± 0.24%) -0.01s (- 0.21%) 5.29s 5.35s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-210-generic
Architecturex64
Available Memory16 GB
Available Memory1 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v10.16.3, x64)
  • Compiler-Unions - node (v12.1.0, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v10.16.3, x64)
  • material-ui - node (v12.1.0, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v10.16.3, x64)
  • xstate - node (v12.1.0, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 49156 10
Baseline main 10

Developer Information:

Download Benchmark

@sandersn sandersn added the Experiment A fork with an experimental idea which might not make it into master label May 23, 2022
@sheetalkamat
Copy link
Member Author

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 9, 2022

Heya @sheetalkamat, I've started to run the tarball bundle task on this PR at 1be59b2. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jun 9, 2022

Hey @sheetalkamat, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/127785/artifacts?artifactName=tgz&fileId=46642C0DDD3E668571B6CCF24513AC8BB9E2EE86E668F55AFAC63EF1F15D9EDE02&fileName=/typescript-4.8.0-insiders.20220609.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

@sheetalkamat
Copy link
Member Author

@DanielRosenwasser I am having hard time reasoning about this change.
There are three places currently doing declaration emit/transforms:

  1. Builder: To get signature
  2. DeclarationDiagnostics: Does only transform and gets error
  3. Actual d.ts emit

As part of this i am caching either transform or text (preferring text over transform) and declaration diagnostics so we can reuse it.
I am unable to see lot of benefit of this though. Mainly because:

  1. By default when we everything is clean and not emitted, the first run: Builder is not going to do signature emit any more (Do not calculate signatures if old state is not used #43314)
  2. Change to store signature computed during emit helps with not having big working set for the next iteration: (During emit, if shape signature for the file is same as version, then update it with emitted d.ts file #48616) so this makes it as if in first build we did d.ts emit so number of files on which d.ts emit would be done to calculate their signature by the builder should be smaller. So perf win is possible only in next edits but that is not huge since the file list is smaller.
  3. noEmitOnError is only when we really get declaration diagnostics separately so thats the only place we would save d.ts transform cost.

I havent been able to find repo where this benefits significantly. Numbers remain more or less same.

@sheetalkamat
Copy link
Member Author

cc @amcasey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team Experiment A fork with an experimental idea which might not make it into master For Milestone Bug PRs that fix a bug with a specific milestone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Initial build mode build emits twice
3 participants