Support for pre-compilation & post-compilation build scripts in tsconfig.json

[jahorton]

Suggestion

🔍 Search Terms

preprocessor, postprocessor, shell script, external script, precompilation, postcompilation, precompile, postcompile, worker, WebWorker, web worker

Obviously the first term matched a lot of issues, but those were more in terms of preprocessor directives, not toolchain hooks.

✅ Viability Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.
  • Possible conflict on non-goal 4, as it can be worked around (somewhat clumsily) in part by use of external tooling.
  • However, workarounds cannot overcome significant limitations for --watch builds without this feature.

⭐ Suggestion

New, optional configuration options in tsconfig.json:

  "compilerOptions": {
    "prebuild": "<external, terminal command>",
    "postbuild": "<external, terminal command>",
  }

Example:

  "compilerOptions": {
    "prebuild": "./write_time-of-build.ts.sh"
  }

or

  "compilerOptions": {
    "postbuild": "./wrap-worker-code.sh"
  }

The first toy example would use a shell script to write time-of-build.ts, a file defining the exact UTC time that the build command was run. This could then be used as part of the build itself, allowing the built product to include its contents and output this as part of an about "page"/"element".

The second example... well, that's the primary motivation for this proposal.

📃 Motivating Example

Now for a meaty case: bundled WebWorker code.

tl;dr: multi-level composite projects exist that can benefit from incremental builds, --watch, etc. Due to library limitations, one level must be built first, and a little bit of tsconfig.json-triggered automatic external tooling would make all the difference for linking the two tiers together while still compatible with tsc --watch and tsc -b.

Now, for the long-form explanation:

We have a group of projects collectively targeting the standard DOM, and we've been in the process of converting to use of a composite-project configuration scheme. (For further context, our codebase still uses namespaces, and we bundle-compile the main top-level products with outFile.) The WebWorker's code is, conceptually, a mid-level product - it utilizes some lower-level projects and is most directly used by a 'wrapper' project designed to provide a DOM-friendly API for communication with the worker, handling all of the needed postMessage and onMessage patterns internally.

The issue we ran into: how do we bundle WebWorker-space code with DOM-space code? We took the 'standard' approach, so far as we know:

Steps for WebWorker bundling

1. You embed the WebWorker code in a wrapping function that preserves the JS (read: TS build product).
2. Embed the wrapping function into the project that will construct the worker.
3. <wrapped function>.toString() allows you to retrieve the original JS "source" (read: TS build product) at run-time.
4. Trim off the wrapper components.
5. Construct a blob from the now-unwrapped source.
6. Pass that constructed (text) blob as the source file for a WebWorker.

For a minimum-repro example of this approach, refer to this example code. It uses tsc -p rather than -b, but that won't affect its validity as a reference. A little effort can convert it right over; it was written pre-3.7. (It's what we used to validate the approach before adopting it on a larger scale.)

As far as raw JS goes, the approach works swimmingly. That said, both our DOM projects and the WebWorker projects are TS, so it'd be nice if there were a way forward for this in tsc. However, as far as compilation goes, tsc will throw an absolute fit trying to compile the two together directly because the two scenarios require very different library inclusions. We recognize that this is a very natural and likely very necessary limitation, so it's reasonable to require some sort of workaround. (For reference, before converting to composite builds, we always just included the lower-level build products - raw JS - and used a lot of shell script tooling to manage everything.) Hence, separate projects.

WebWorker project:

  "compilerOptions": {
    "lib": ["webworker", "es6"]
  }

The worker-interfacing project:

  "compilerOptions": {
    "lib": ["es6", "dom"]
  }

If we could reasonably author all of the WebWorker's code within a single file, maybe the wrapping aspect wouldn't be a big issue... but there's a lot of code, and so it's a multi-file project. (Multiple classes, etc. Monolithic files aren't the best development practice.) As a result, there's no way we can directly build to a single function that wraps all worker related code via tsc. (See steps 1 and 3 above.) That said, outFile does allow us to build a single output file that we can externally wrap. That gives us one important tool to proceed, and it already exists.

So, next we have a dedicated shell script (say, build.sh) that runs tsc (with outFile) and then creates a new file with a wrapped version. The problem... is that for the higher-level projects, we now have a tsconfig "include" file that's a build output from a shell script. We can't "reference" it in consuming tsconfig.json files - we've got that "lib" conflict to avoid, which would block step 2 above. And we want to leverage composite-project TypeScript builds.

In case anyone came here suffering a similar problem, our current workaround: as the worker's build output is in JS, we needed to rename it as a TS file and add @ts-ignore at the top of the wrapping file - otherwise, we'd get problems with a lack of internal type decoration, which would also block step 2 above. After all, composite projects require declaration emit, and you can't emit well from wrapped raw JS. The exact error we got before using @ts-ignore: error TS9005: Declaration emit for this file requires using private name 'xxx'. An explicit type annotation may unblock declaration emit. (One suggestion we received -"jsCheck": false - didn't prevent it.) The wrapping file can then be added as an "include", not a "reference".

Why it matters

This leads to the main issue and the reason for this proposal: because those projects have to reference include the build product, not the WebWorker's project itself, we're required to run its build-wrapping shell-script each time a change is made in that project. --watch can't utilize this. After all... tsc, by itself, doesn't know to check for any changes to the worker's project, as the project is not referenced. All higher level projects will be referencing including a build product of the worker project, not referencing its original source. As a result, the top-level project can't adequately watch for code changes throughout and auto-update. (I'm also curious about implications for incremental builds, though I suppose they could just hash the wrapped output file and note change, or lack thereof, that way?)

We'd really like it if there were a place in tsconfig.json where we could stick a call to that project's build.sh (which builds the wrapped worker) so that the top-level projects can be fully kept up to date from their respective top-level tsc -b composite-build calls. One simple call to a simple shell script (via the proposed tsconfig.json compilerOptions.postBuild option) would be enough to auto-wrap the worker code for consumption by the next project in line, which could then include it. --watch could be configured to watch for changes in that project's files, and that project could, in post-build, run the wrapping script automatically. Consuming projects then automatically receive the updated build product via include, and everything updates appropriately.

I will admit that an alternative approach would be to simply compile a dummy version of the wrapping function into the build product, then after the main tsc build, do a search-and-replace on it (via unique, identifying string) and dynamically insert the worker code in then. That said, this approach still suffers the same issue - now we need a post-build script at the top level, rather than a pre-build script at the mid-level. (And it's less "encapsulated", as far as the projects go. We prefer wrapping the worker code in the project that directly consumes it if possible.) Even then, I suppose there is some leeway - I don't see why a post-build on the mid-level couldn't be run as part of the top-level's post-build step as well, should a future implementation choose to go that far.

💻 Use Cases

See: motivating example with WebWorkers.

Other cases:

• We currently use a similar approach to build a version.inc.ts file in order to embed variations on our product's current version (major, minor, patch, major.minor.patch, major.minor.patch-releaseTier, etc) into our projects at build time.
  • We find this useful for cross-language, cross-platform semantic versioning. There are also C++, Obj-C, Java, and Swift projects all assigned the same version information at the same time... and they can't rely on package.json version information like a JS/TS project can. (Especially for Mac projects, with their dependence on .plist files.)
    • This, in turn, is very helpful to us for preventing version confusion during customer support interactions.
  • Obviously, this part isn't relevant for the --watch aspects of this proposal, but it does benefit from the same approach.
    • A single pre-build to emit the version info from the common source into a TS file could then be compiled into the project and included in that level and in any higher-level projects.
• The hypothetical "time of build" case.

We've documented our main workaround above in terms of WebWorkers, but it also works pretty well for the version.inc.ts case; while there's no need to "wrap" raw JS output in TS, we still need to call a script to write TS - the dynamically-generated version .ts file - for inclusion in the projects that use it. (So, a pre-tsc write_version_info.sh step.) Obviously, this requires the main driver of the build to be a shell script that first runs the proposed "precompile" step, then runs the main tsc build.

Can that be achieved by external tools? Yes - relying on shell scripts that use tsc builds is enough for main builds of the project.

However, you lose the ability for --watch to auto-build updates when needed, which particularly matters for the WebWorker scenario. Consider how this ties into coverage for --watch-based unit testing solutions -- integration-style tests will not receive code updates. We do have a few defined integration-style tests for our projects - tests to ensure that an outer project is able to communicate with and receive expected data from the eventual WebWorker.

💭 Other notes

Obviously, nomenclature and the exact manner of solution may be changed to best fit the vision of the TS team; this is simply the best way I know to explain the problem in a relatively straightforward and clear manner. In writing this up, I realize I may be wanting composite builds to be... "a little more than they are", so to speak... but they do appear to be the best tool for the job right now, despite their limitations in this matter.

The core essence of our issue probably arises from the fact that we want to distribute a DOM project that depends on a WebWorker project in a single output-file bundle. tsc can't do hybrid-lib builds, so we're forced to stage the build into 'layers'. However, this layering blocks the ability of --watch to cope and properly watch anything at a 'lower level' of the build, so a full project --watch is currently impossible.

It may or may not also affect aspects of incremental building? But that part's just a guess on my end.

[fatcerberus]

I can't find the relevant comment now but ideas like this have been rejected in the past because it would make just running tsc on an unknown codebase potentially unsafe.

[jahorton]

Oh, one more thing: on a related note, source mapping for the Worker is a royal pain to try establishing; I haven't figured it out for this single-file bundling approach, and it'd likely require something custom on my end at present. So...

1. If someone knows of a tool to rejig the sourcemap or provide it within a secondary blob when inspecting the Worker, that'd be pretty rad.
2. If a different approach were possible that addresses the above concerns and also allows tsc-handled sourcemaps for the bundled WebWorker, that'd be absolutely amazing.
   • I do realize that the approach above almost certainly wouldn't handle the worker sourcemap issue, of course.
   • This has been a pain point for us when debugging worker code; among other things, we have a generator function in TS, and that does not compile down very intuitively or elegantly to ES5. Definitely not fun to debug that tidbit without available sourcemapping.

[RyanCavanaugh]

I'd really like to approach this scenario by inverting the relationship between the build scripts and TypeScript.

Our hesitation to add something like this is based on a lot of different potential issues:

• The exit code of these scripts presumably matters, so determining what to do when a prebuild fails (or worse, when a postbuild fails) is a giant can of policy worms
• Bare tsc in a repo is "supposed to" be a safe operation in terms of not running any untrusted code. This is in-principle addressable with forcing you to write tsc --run-scripts-dangerous or something, but at that point now the tsc invocation is arbitrarily annoying and will be put into an npm run tsc wrapper, at which point you may as well be invoking some larger orchestrating tool that has more policy information available to you
• Immediately, people will be annoyed that editors don't execute prebuild on their behalf, since it's likely that there will be errors in their code until this happens (e.g. because their schema -> .d.ts generator runs on prebuild). But we can't run prebuild automatically from editor scenarios because it's a security problem. This is already a problem in composite builds -- we hear feedback from partner teams that having people do something even just once on repo clone is too burdensome (you might be skeptical of this feedback, and I am too, but there it is)
• Both scripts might have non-TS input files which watch won't be able to know about, so there's an immediate follow-up feature to add some interplay between the tsc watching process and the build scripts so that a watch build is "always up to date"
• If the postbuild script modifies an input file, watch might get into an infinite loop, which we'll be blamed for
• Architecturally, we compute the entire file list prior while parsing the tsconfig file, so if prebuild produces a new file on disk, we'll "miss" it. We could rewrite this logic but it'd potentially be an API breaking change
• These are just the ones I came up with off the top of my head, so there are certainly more

Good alternative options I see would be:

• Some platform-agnosticish way to "signal" that a watch mode build is complete, plus another way to say "block watch rebuild until X signal happens". An orchestration process can start tsc -w and the prebuild/postbuild runner and coordinate them
• Make the watch API more friendly so that it can be hosted from a script
• Something else?