Slow compilation with really huge declaration files

[PanayotCankov]

Hello,

The NativeScript framework exposes all native Android and iOS APIs to JavaScript. To take use of the TypeScript features we have tools to generate TypeScript declaration files. These declaration files are verified to have no syntax or type errors when they are generated.

Compiling a small ts file that references types from the big d.ts-es results in about 15 sec. compilation time on our side. Anything that can help us to speed up the build time would be greatly appreciated.

This is how the diagnostics of a project using our .d.ts-es looks like:

nativescript-big-dts cankov$ tsc
Files:               4
Lines:          317792
Nodes:         1235680
Identifiers:    486912
Symbols:        709471
Types:          145659
Memory used:   824485K
I/O read:        0.02s
I/O write:       0.00s
Parse time:      2.04s
Bind time:       1.32s
Check time:     12.09s
Emit time:       0.03s
Total time:     15.49s

I have prepared a small project suitable for profiling here:
https://github.com/PanayotCankov/nativescript-big-dts

Thanks,

[s-panferov]

@PanayotCankov please take a look at the similar discussion here #8082

[PanayotCankov]

We would be fine if these big .d.ts-es are not type checked.

If there is concern that other sources may invalidate them - we would be fine if the types in ios.d.ts and android.d.ts are self-contained (use only primitive types and do not use types from other sources) and “closed” (so they can’t be extended and thus invalidated from outside).

[mhegazy]

please note it is not just typecjecking. it is all tools that have to deal with this project. first the large file needs to be parsed, and kept in memory, the tools need to do refactoring on it, etc..

the best solution is to see if you can break this file into smaller pieces, and only include required pieces. anything else will just cause another problem.

[PanayotCankov]

Hi @mhegazy,

Everyone's first suggestion is to split the files. We can split the iOS APIs by frameworks right? Then iOS's "categories" kick in and requiring the UIKit pulls most of the iOS SDK so we end up with very little benefit. Then we start talking how we can use extension methods or try to declare the classes as decomposed somehow to limit the dependencies between the frameworks or cut some of the methods just to avoid UIKit pulling AVFoundation, Quartz and the rest of the rather rarely used framework. Then we put that in real app only to find out that some UI and IO operations require enough of the iOS frameworks just to make the build annoyingly slow. Meanwhile the project gets full of imports or /// <reference path=""/> and cripple the tools, that initially worked somewhat fine (given a dev machine has 16+ GB of ram anyway) so we end up combining back to the state in the referenced repo so we can take full use of the tooling, simply complaining from the build time and nothing else.

If you could do anything to address the 800MB memory used to represent the type information in 15MB header files it would be nice but if you could make anything to speed up the build of this project it would be totally awesome.

I do not insist on skipping type checking. It is just the slowest thing in the diagnostics.

I would prefer if the .d.ts are locked and can not be refactored. There is no way we could reflect a change in the .d.ts back in the native SDKs. I believe such is the case with @s-panferov's db.

[mhegazy]

refactoring symbols in a .d.ts is tracked by #7458 (comment).

I think we are saying the same thing. the main issue is the size of the declaration file. we keep AST in memory, symbols and types. these files are packed with types, and they all live in the global namespace (similar to lib.d.ts). given that size, you hit a way fairly quickly. not type checking them can save sometime, but does not address the underlying issue, which is that is a lot of definitions to include for a hello-world-type program. Would love to hear your suggestions on how to make that better.

[PanayotCankov]

My first suggestion would be to lazy parse / type check only what is used in a client's app.

This is the same that happens if we separate the big files in small ones and let the clients pick few of them. However this does not put some of the limits such as us filtering some of the APIs to limit the cross-framework dependencies.

So does it seem reasonable to scan the source .d.ts file initially, get map with potential types but store just the position in the source, don't even try to build AST, then if such lazy type is used somewhere else then do the complete parsing and type checking. Work with the assumption that a .d.ts by itself has no errors but do type-checking where errors may be introduced e.g when client implements a class, or when two .d.ts-es add members to the same interface etc.

Internally it sounds a good idea to build a script based on the compiler API to ignore type checking in the big .d.ts files so we could speed up development time. Our CI will still run full checks so we can have the type safety for PRs.

[evmar]

I was also going to suggest using tsc -w, but when I tried that it didn't seem to help much. If you build once, then add a line to the tiny input .ts file, the compiler prints "File change detected. Starting incremental compilation..." but it takes 10 seconds to build. From the printed stats I guess that avoids the parse/bind time, but the bulk of the time is in "check".

The android.d.ts file is nearly 10x larger than the iOS one. The very first class you see in it if you open it is a class marked "Legacy security code; do not use." If this were split into modules that matched the Java modules you'd likely see a speedup. If you take it as a given that the iOS headers are unfactorable (must be one large blob) you can still get your incremental build time down to ~3s with separate Android modules and the above trick.

[mhegazy]

For modules we can do better, in fact we have an issues to optimize tsc --w with modules (see #3204). for global code it is not clear what we can do and still be type safe.

[mhegazy]

@PanayotCankov deferred parsing is definitely a solution, but it is a huge undertaking. the compiler as it i structured today, assumes a phased approach; supporting something like this would require a complete restructuring of the code base.

[PanayotCankov]

@mhegazy I think we can go with isolatedModules, {N} uses CommonJS modules for the JavaScript code. We considered using external ambient module declarations for the native APIs and may expose the native APIs through the module system (e.g. import { UIView } from "objc!UIKit";).

We were waiting for "extension methods" to separate the iOS by frameworks but as it turns out this is already possible.

[PanayotCankov]

Event when we resolve some of the dependencies from methods in iOS categories, in Objective-C there are still forward declarations for classes in other frameworks. In the end we can limit the dependencies a bit.

What we could do by separating frameworks is to reach Total time: 3.52s for 12 files referenced in UIKit compared to Total time: 7.49s when building all 88 files. UIKit is BIG.

[PanayotCankov]

@martine The android.d.ts is 10x larger because of the docs (/** */), Android's license was permissive enough to let us extract the Doc comments from the Java source code. Otherwise the types are somewhat more than in iOS but the difference is not that big.

Btw if I remember correctly the build speeds up if we remove the comments.

[mhegazy]

So after going back and forth on this, we will experiment with adding a new switch to disable checking for .d.ts files. hopefully this gives you some relief. see more relevant discussion in #8724

[PanayotCankov]

Thanks guys,

The good part is that playing with such big files at least shows the compiler scales well and the compilation time in our case is somewhat linear based on the amount of source code.

We've been using TS since it was 0.95 and it proved to be great.
TypeScript is one of those things that make Microsoft cool nowadays.

[mhegazy]

thanks for the kind words. please do let us know if you run into other issues.