Skip to content

Don't skip elaborations when reporting errors for cached failed relations #55234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

Andarist
Copy link
Contributor

@Andarist Andarist commented Aug 1, 2023

fixes #3276
to learn more about what's happening you can read my comment here

I'm just opening this as a draft, even without updating baselines. It might be interesting to run the perf suite on this one.

@typescript-bot typescript-bot added the For Backlog Bug PRs that fix a backlog bug label Aug 1, 2023
@RyanCavanaugh
Copy link
Member

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 2, 2023

Heya @RyanCavanaugh, I've started to run the abridged perf test suite on this PR at eaba5a1. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@RyanCavanaugh
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..55234

Metric main 55234 Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 368,212k (± 0.00%) 368,215k (± 0.00%) ~ 368,195k 368,243k p=0.936 n=6
Parse Time 3.56s (± 0.63%) 3.56s (± 0.63%) ~ 3.55s 3.61s p=1.000 n=6
Bind Time 1.19s (± 0.34%) 1.19s (± 0.43%) ~ 1.18s 1.19s p=0.114 n=6
Check Time 9.73s (± 0.54%) 9.71s (± 0.18%) ~ 9.69s 9.73s p=0.935 n=6
Emit Time 8.00s (± 0.86%) 7.95s (± 0.49%) ~ 7.88s 7.99s p=0.377 n=6
Total Time 22.48s (± 0.53%) 22.42s (± 0.25%) ~ 22.31s 22.47s p=0.689 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,820k (± 0.01%) 193,832k (± 0.02%) ~ 193,780k 193,880k p=0.689 n=6
Parse Time 1.59s (± 0.56%) 1.58s (± 1.31%) ~ 1.55s 1.61s p=0.804 n=6
Bind Time 0.83s (± 1.25%) 0.82s (± 0.77%) ~ 0.81s 0.83s p=0.203 n=6
Check Time 10.13s (± 0.39%) 10.13s (± 0.54%) ~ 10.06s 10.22s p=0.872 n=6
Emit Time 3.01s (± 1.38%) 2.98s (± 0.78%) ~ 2.94s 3.01s p=0.258 n=6
Total Time 15.56s (± 0.41%) 15.51s (± 0.51%) ~ 15.39s 15.62s p=0.423 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,071k (± 0.01%) 347,072k (± 0.00%) ~ 347,051k 347,085k p=1.000 n=6
Parse Time 2.78s (± 0.63%) 2.76s (± 0.48%) ~ 2.74s 2.78s p=0.143 n=6
Bind Time 1.08s (± 0.59%) 1.07s (± 0.38%) -0.01s (- 0.77%) 1.07s 1.08s p=0.033 n=6
Check Time 8.03s (± 0.38%) 8.01s (± 0.46%) ~ 7.94s 8.04s p=0.373 n=6
Emit Time 4.47s (± 0.45%) 4.46s (± 0.87%) ~ 4.41s 4.51s p=0.744 n=6
Total Time 16.36s (± 0.24%) 16.30s (± 0.33%) ~ 16.24s 16.37s p=0.107 n=6
TFS - node (v16.17.1, x64)
Memory used 301,124k (± 0.01%) 301,114k (± 0.01%) ~ 301,081k 301,142k p=0.630 n=6
Parse Time 2.20s (± 0.60%) 2.19s (± 0.63%) ~ 2.18s 2.21s p=0.805 n=6
Bind Time 1.22s (± 0.96%) 1.20s (± 0.43%) -0.02s (- 1.78%) 1.19s 1.20s p=0.004 n=6
Check Time 7.32s (± 0.26%) 7.31s (± 0.24%) ~ 7.28s 7.33s p=0.415 n=6
Emit Time 4.31s (± 0.91%) 4.32s (± 0.56%) ~ 4.28s 4.35s p=0.686 n=6
Total Time 15.04s (± 0.37%) 15.01s (± 0.32%) ~ 14.95s 15.09s p=0.296 n=6
material-ui - node (v16.17.1, x64)
Memory used 481,871k (± 0.01%) 481,880k (± 0.01%) ~ 481,821k 481,921k p=0.873 n=6
Parse Time 3.25s (± 0.36%) 3.25s (± 0.49%) ~ 3.24s 3.28s p=0.863 n=6
Bind Time 0.96s (± 0.78%) 0.95s (± 1.10%) ~ 0.94s 0.97s p=0.611 n=6
Check Time 18.30s (± 0.64%) 18.32s (± 0.41%) ~ 18.24s 18.43s p=0.375 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.50s (± 0.48%) 22.53s (± 0.36%) ~ 22.43s 22.63s p=0.423 n=6
xstate - node (v16.17.1, x64)
Memory used 561,397k (± 0.01%) 561,338k (± 0.02%) ~ 561,247k 561,540k p=0.298 n=6
Parse Time 4.04s (± 0.34%) 4.01s (± 0.20%) -0.02s (- 0.54%) 4.00s 4.02s p=0.009 n=6
Bind Time 1.70s (± 6.37%) 1.70s (± 4.91%) ~ 1.58s 1.78s p=0.747 n=6
Check Time 3.12s (± 2.88%) 3.08s (± 2.87%) ~ 3.02s 3.25s p=0.147 n=6
Emit Time 0.09s (± 4.45%) 0.09s (± 5.53%) ~ 0.09s 0.10s p=0.595 n=6
Total Time 8.94s (± 0.59%) 8.89s (± 0.79%) ~ 8.77s 8.98s p=0.261 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-148-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current 55234 6
Baseline main 6

Developer Information:

Download Benchmark

@Andarist Andarist force-pushed the dont-skip-elaboration-on-cached-entries branch from 2bf3cf9 to 4a88fbd Compare August 6, 2023 19:51
@Andarist
Copy link
Contributor Author

Andarist commented Aug 6, 2023

@RyanCavanaugh since the perf run came up clean I updated the baselines and cleaned up this a little bit further. Looking at the changed baselines it seems like a neat improvement to me :)

@weswigham
Copy link
Member

@typescript-bot test this
@typescript-bot run dt
@typescript-bot test top100
@typescript-bot new perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 14, 2023

Heya @weswigham, I've started to run the diff-based top-repos suite on this PR at 4a88fbd. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 14, 2023

Heya @weswigham, I've started to run the regular perf test suite on this PR at 4a88fbd. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 14, 2023

Heya @weswigham, I've started to run the parallelized Definitely Typed test suite on this PR at 4a88fbd. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@weswigham
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 300,343k (± 0.00%) 300,351k (± 0.00%) ~ 300,343k 300,360k p=0.064 n=6
Parse Time 3.03s (± 0.13%) 3.03s (± 0.17%) ~ 3.02s 3.03s p=0.595 n=6
Bind Time 0.93s (± 0.44%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=0.405 n=6
Check Time 9.45s (± 0.20%) 9.45s (± 0.18%) ~ 9.43s 9.47s p=0.685 n=6
Emit Time 7.62s (± 0.35%) 7.64s (± 0.18%) ~ 7.62s 7.66s p=0.512 n=6
Total Time 21.04s (± 0.16%) 21.05s (± 0.06%) ~ 21.03s 21.07s p=0.410 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,891k (± 0.01%) 193,879k (± 0.01%) ~ 193,865k 193,899k p=0.575 n=6
Parse Time 1.58s (± 0.26%) 1.57s (± 0.26%) -0.01s (- 0.42%) 1.57s 1.58s p=0.034 n=6
Bind Time 0.79s (± 0.00%) 0.79s (± 0.00%) ~ 0.79s 0.79s p=1.000 n=6
Check Time 9.93s (± 0.56%) 9.91s (± 0.46%) ~ 9.84s 9.97s p=0.520 n=6
Emit Time 2.75s (± 0.27%) 2.74s (± 0.30%) ~ 2.73s 2.75s p=0.383 n=6
Total Time 15.04s (± 0.37%) 15.02s (± 0.26%) ~ 14.96s 15.07s p=0.422 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,031k (± 0.01%) 347,036k (± 0.01%) ~ 347,022k 347,068k p=0.336 n=6
Parse Time 2.69s (± 0.19%) 2.69s (± 0.38%) ~ 2.67s 2.70s p=0.928 n=6
Bind Time 0.99s (± 0.00%) 0.99s (± 0.41%) ~ 0.98s 0.99s p=0.405 n=6
Check Time 7.90s (± 0.49%) 7.89s (± 0.26%) ~ 7.85s 7.91s p=0.629 n=6
Emit Time 4.27s (± 0.76%) 4.28s (± 0.32%) ~ 4.26s 4.30s p=0.331 n=6
Total Time 15.84s (± 0.43%) 15.84s (± 0.18%) ~ 15.79s 15.87s p=0.748 n=6
TFS - node (v16.17.1, x64)
Memory used 301,106k (± 0.00%) 301,112k (± 0.00%) ~ 301,104k 301,118k p=0.469 n=6
Parse Time 2.16s (± 0.45%) 2.16s (± 0.54%) ~ 2.15s 2.18s p=1.000 n=6
Bind Time 1.11s (± 0.00%) 1.11s (± 0.00%) ~ 1.11s 1.11s p=1.000 n=6
Check Time 7.23s (± 0.36%) 7.20s (± 0.20%) ~ 7.19s 7.23s p=0.088 n=6
Emit Time 3.97s (± 0.19%) 3.98s (± 0.52%) ~ 3.96s 4.01s p=0.209 n=6
Total Time 14.47s (± 0.19%) 14.46s (± 0.29%) ~ 14.41s 14.53s p=0.373 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,372k (± 0.00%) 479,368k (± 0.00%) ~ 479,356k 479,397k p=0.378 n=6
Parse Time 3.15s (± 0.13%) 3.15s (± 0.47%) ~ 3.14s 3.18s p=1.000 n=6
Bind Time 0.91s (± 0.00%) 0.91s (± 0.00%) ~ 0.91s 0.91s p=1.000 n=6
Check Time 17.89s (± 0.31%) 17.90s (± 0.49%) ~ 17.77s 18.03s p=0.936 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.95s (± 0.25%) 21.96s (± 0.42%) ~ 21.83s 22.09s p=1.000 n=6
xstate - node (v16.17.1, x64)
Memory used 542,764k (± 0.01%) 542,800k (± 0.02%) ~ 542,710k 542,996k p=0.873 n=6
Parse Time 3.69s (± 0.14%) 3.69s (± 0.14%) ~ 3.69s 3.70s p=1.000 n=6
Bind Time 1.46s (± 0.28%) 1.46s (± 0.00%) ~ 1.46s 1.46s p=0.405 n=6
Check Time 3.16s (± 0.35%) 3.17s (± 0.26%) +0.01s (+ 0.42%) 3.16s 3.18s p=0.029 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.405 n=6
Total Time 8.39s (± 0.16%) 8.41s (± 0.17%) ~ 8.39s 8.43s p=0.060 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,475ms (± 0.07%) 2,476ms (± 0.18%) ~ 2,469ms 2,483ms p=0.517 n=6
Req 2 - geterr 5,956ms (± 0.40%) 5,955ms (± 0.67%) ~ 5,914ms 6,029ms p=0.936 n=6
Req 3 - references 340ms (± 0.64%) 340ms (± 0.71%) ~ 338ms 344ms p=1.000 n=6
Req 4 - navto 279ms (± 0.43%) 279ms (± 0.55%) ~ 277ms 280ms p=0.923 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 84ms (± 2.70%) 85ms (± 4.20%) ~ 82ms 90ms p=0.507 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,610ms (± 1.02%) 2,609ms (± 0.87%) ~ 2,582ms 2,636ms p=0.936 n=6
Req 2 - geterr 4,721ms (± 0.18%) 4,728ms (± 0.29%) ~ 4,705ms 4,743ms p=0.230 n=6
Req 3 - references 350ms (± 0.15%) 350ms (± 0.23%) ~ 349ms 351ms p=0.929 n=6
Req 4 - navto 269ms (± 0.28%) 269ms (± 0.28%) ~ 268ms 270ms p=0.487 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 79ms (± 0.00%) 79ms (± 0.52%) ~ 78ms 79ms p=0.405 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,717ms (± 0.26%) 2,717ms (± 0.29%) ~ 2,703ms 2,725ms p=1.000 n=6
Req 2 - geterr 1,947ms (± 1.84%) 1,952ms (± 0.37%) ~ 1,941ms 1,959ms p=0.196 n=6
Req 3 - references 126ms (±10.44%) 129ms (± 8.41%) ~ 116ms 143ms p=0.419 n=6
Req 4 - navto 354ms (± 0.64%) 352ms (± 0.43%) ~ 350ms 354ms p=0.103 n=6
Req 5 - completionInfo count 2,071 (± 0.00%) 2,071 (± 0.00%) ~ 2,071 2,071 p=1.000 n=6
Req 5 - completionInfo 317ms (± 0.83%) 319ms (± 1.58%) ~ 315ms 329ms p=0.934 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • CompilerTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 155.34ms (± 0.18%) 155.28ms (± 0.15%) -0.06ms (- 0.04%) 154.46ms 159.44ms p=0.013 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 230.59ms (± 0.14%) 230.62ms (± 0.13%) ~ 229.67ms 236.65ms p=0.535 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 234.25ms (± 0.17%) 235.20ms (± 0.14%) +0.95ms (+ 0.41%) 233.81ms 237.83ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 217.65ms (± 0.13%) 217.55ms (± 0.13%) -0.10ms (- 0.04%) 216.26ms 222.65ms p=0.001 n=600
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@typescript-bot
Copy link
Collaborator

@weswigham Here are the results of running the top-repos suite comparing main and refs/pull/55234/merge:

Everything looks good!

@typescript-bot
Copy link
Collaborator

Hey @weswigham, the results of running the DT tests are ready.
Everything looks the same!
You can check the log here.

@sandersn
Copy link
Member

@weswigham @ahejlsberg I'd like to merge this at the start of 5.4 development. Is it good to go?

Comment on lines 20932 to +20934
// Record this relation as having failed such that we don't attempt the overflowing operation again.
const id = getRelationKey(source, target, /*intersectionState*/ IntersectionState.None, relation, /*ignoreConstraints*/ false);
relation.set(id, RelationComparisonResult.Reported | RelationComparisonResult.Failed);
relation.set(id, RelationComparisonResult.Failed);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sandersn @ahejlsberg those are lines that were not at all here when I created this PR. When syncing with main now I removed the RelationComparisonResult.Reported - just like it was done everywhere else here but I'm not sure if this should actually be done here.

Those lines come from #55851

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, the CI shows that this actually is important as now it's doing a lot of heavy work twice in this pathological case and the CI timeouts.

If we take a closer look at this baseline (even on main) we might notice that it's actually reporting 2 errors at the same location:

relationComplexityError.ts(12,5): error TS2322: Type 'T1 & T2' is not assignable to type 'T1 | null'.
relationComplexityError.ts(12,5): error TS2859: Excessive complexity comparing types 'T1 & T2' and 'T1 | null'.

So perhaps If I manage to report this once, the perf problems will go away naturally.

@sandersn
Copy link
Member

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Nov 30, 2023

Heya @sandersn, I've started to run the regular perf test suite on this PR at 0611fbb. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@sandersn
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,301k (± 0.01%) 295,288k (± 0.01%) ~ 295,268k 295,325k p=0.173 n=6
Parse Time 2.64s (± 0.31%) 2.65s (± 0.37%) ~ 2.64s 2.67s p=0.177 n=6
Bind Time 0.82s (± 0.00%) 0.82s (± 0.50%) ~ 0.82s 0.83s p=0.405 n=6
Check Time 8.05s (± 0.22%) 8.06s (± 0.21%) ~ 8.04s 8.08s p=0.288 n=6
Emit Time 7.08s (± 0.30%) 7.08s (± 0.23%) ~ 7.06s 7.09s p=0.675 n=6
Total Time 18.59s (± 0.10%) 18.61s (± 0.13%) ~ 18.58s 18.64s p=0.122 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 195,142k (± 1.55%) 194,070k (± 1.62%) ~ 191,169k 196,952k p=0.092 n=6
Parse Time 1.36s (± 0.90%) 1.36s (± 1.11%) ~ 1.34s 1.38s p=1.000 n=6
Bind Time 0.72s (± 0.57%) 0.72s (± 0.00%) ~ 0.72s 0.72s p=0.405 n=6
Check Time 9.25s (± 0.22%) 9.20s (± 0.47%) -0.05s (- 0.52%) 9.14s 9.27s p=0.043 n=6
Emit Time 2.64s (± 0.40%) 2.59s (± 0.20%) -0.04s (- 1.58%) 2.59s 2.60s p=0.004 n=6
Total Time 13.96s (± 0.13%) 13.87s (± 0.29%) -0.09s (- 0.62%) 13.83s 13.94s p=0.008 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,358k (± 0.00%) 347,348k (± 0.01%) ~ 347,312k 347,368k p=0.574 n=6
Parse Time 2.46s (± 0.49%) 2.46s (± 0.47%) ~ 2.44s 2.47s p=0.502 n=6
Bind Time 0.92s (± 0.82%) 0.93s (± 0.44%) ~ 0.92s 0.93s p=0.100 n=6
Check Time 6.91s (± 0.53%) 6.90s (± 0.52%) ~ 6.84s 6.94s p=0.935 n=6
Emit Time 4.06s (± 0.64%) 4.05s (± 0.45%) ~ 4.02s 4.07s p=0.568 n=6
Total Time 14.35s (± 0.36%) 14.34s (± 0.43%) ~ 14.22s 14.39s p=0.810 n=6
TFS - node (v18.15.0, x64)
Memory used 302,656k (± 0.01%) 302,634k (± 0.01%) ~ 302,620k 302,664k p=0.470 n=6
Parse Time 2.00s (± 0.80%) 2.01s (± 1.04%) ~ 1.99s 2.05s p=0.154 n=6
Bind Time 1.00s (± 1.36%) 1.01s (± 1.32%) ~ 0.99s 1.02s p=0.550 n=6
Check Time 6.26s (± 0.22%) 6.25s (± 0.39%) ~ 6.21s 6.28s p=0.682 n=6
Emit Time 3.59s (± 0.38%) 3.59s (± 0.35%) ~ 3.58s 3.61s p=0.868 n=6
Total Time 12.84s (± 0.18%) 12.85s (± 0.21%) ~ 12.81s 12.89s p=0.406 n=6
material-ui - node (v18.15.0, x64)
Memory used 506,728k (± 0.01%) 506,722k (± 0.01%) ~ 506,676k 506,752k p=0.873 n=6
Parse Time 2.58s (± 0.49%) 2.58s (± 0.45%) ~ 2.57s 2.60s p=0.931 n=6
Bind Time 0.99s (± 0.76%) 0.99s (± 1.23%) ~ 0.98s 1.01s p=0.388 n=6
Check Time 16.78s (± 0.23%) 16.72s (± 0.47%) ~ 16.60s 16.82s p=0.128 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.35s (± 0.18%) 20.29s (± 0.36%) ~ 20.18s 20.37s p=0.199 n=6
xstate - node (v18.15.0, x64)
Memory used 512,886k (± 0.01%) 512,828k (± 0.02%) ~ 512,736k 512,987k p=0.199 n=6
Parse Time 3.27s (± 0.25%) 3.28s (± 0.12%) ~ 3.27s 3.28s p=0.248 n=6
Bind Time 1.53s (± 0.36%) 1.54s (± 0.34%) ~ 1.53s 1.54s p=0.640 n=6
Check Time 2.81s (± 0.37%) 2.81s (± 0.54%) ~ 2.79s 2.83s p=0.934 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.405 n=6
Total Time 7.70s (± 0.15%) 7.71s (± 0.28%) ~ 7.68s 7.74s p=0.334 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,337ms (± 0.45%) 2,340ms (± 0.36%) ~ 2,330ms 2,353ms p=1.000 n=6
Req 2 - geterr 5,467ms (± 1.46%) 5,444ms (± 1.72%) ~ 5,333ms 5,539ms p=0.936 n=6
Req 3 - references 328ms (± 1.89%) 329ms (± 1.58%) ~ 323ms 336ms p=1.000 n=6
Req 4 - navto 276ms (± 1.27%) 277ms (± 1.27%) ~ 273ms 280ms p=0.734 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 87ms (± 5.09%) 85ms (± 6.91%) ~ 76ms 90ms p=0.804 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,474ms (± 0.89%) 2,480ms (± 0.65%) ~ 2,459ms 2,499ms p=0.336 n=6
Req 2 - geterr 4,113ms (± 2.05%) 4,077ms (± 1.61%) ~ 4,026ms 4,207ms p=0.936 n=6
Req 3 - references 339ms (± 1.52%) 342ms (± 1.44%) ~ 334ms 346ms p=0.418 n=6
Req 4 - navto 284ms (± 0.36%) 284ms (± 0.43%) ~ 283ms 286ms p=1.000 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 83ms (± 7.55%) 86ms (± 6.45%) ~ 79ms 90ms p=0.279 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,598ms (± 0.74%) 2,600ms (± 0.68%) ~ 2,571ms 2,625ms p=0.936 n=6
Req 2 - geterr 1,669ms (± 1.80%) 1,668ms (± 1.95%) ~ 1,636ms 1,717ms p=1.000 n=6
Req 3 - references 115ms (± 8.39%) 115ms (± 8.71%) ~ 103ms 124ms p=0.870 n=6
Req 4 - navto 364ms (± 1.01%) 368ms (± 1.40%) ~ 365ms 378ms p=0.618 n=6
Req 5 - completionInfo count 2,073 (± 0.00%) 2,073 (± 0.00%) ~ 2,073 2,073 p=1.000 n=6
Req 5 - completionInfo 308ms (± 1.31%) 308ms (± 1.44%) ~ 305ms 316ms p=1.000 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.06ms (± 0.21%) 152.73ms (± 0.18%) -0.33ms (- 0.21%) 151.61ms 156.15ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.13ms (± 0.17%) 227.64ms (± 0.15%) -0.49ms (- 0.22%) 226.26ms 231.15ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 229.40ms (± 0.19%) 229.25ms (± 0.17%) -0.15ms (- 0.06%) 227.61ms 232.08ms p=0.001 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 229.02ms (± 0.19%) 229.08ms (± 0.20%) ~ 227.32ms 236.04ms p=0.155 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@gabritto
Copy link
Member

We had this PR all this time and never merged it? Oh well.

@Andarist
Copy link
Contributor Author

@gabritto does it overlap with some other fresh work/issue? I can certainly try to prioritize this PR more and bring it up to speed

@jakebailey
Copy link
Member

Yeah, #57842, since we may report diagnostics out of order and therefore elaborations may jump around.

@gabritto
Copy link
Member

Yeah, #57842, since we may report diagnostics out of order and therefore elaborations may jump around.

To give more details, I'm doing some diagnostics work in preparation for #57842. Right now I'm trying to have less duplicated diagnostics, but next on my list is to provide assignability elaboration consistently in editor scenarios, and as a consequence get rid of some of the diagnostics differences that arise when using #57842 vs not using them.
Where things might overlap is that the fact that assignability diagnostics varies depending on cached relations sometimes makes it harder to detect duplication, so making the assignability diagnostics more consistent could help with that as well.

@Andarist
Copy link
Contributor Author

This work has landed as part of #58859

@Andarist Andarist closed this Jul 15, 2024
@gabritto
Copy link
Member

This work has landed as part of #58859

Thanks for doing this!

@sandersn sandersn removed this from PR Backlog Apr 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
For Backlog Bug PRs that fix a backlog bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Diagnostic elaboration is not consistently reported in language service
8 participants