feat: support compile and extract with multi thread

[Zxilly]

Description

Offload the task to worker_thread to utilize the computing power of multiple cores.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Examples update

This is a follow up of #2299

Checklist

• I have read the CONTRIBUTING and CODE_OF_CONDUCT docs
• I have added tests that prove my fix is effective or that my feature works
• I have added the necessary documentation (if appropriate)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Updated (UTC)
js-lingui	Ready	Preview	Aug 22, 2025 3:25pm

[timofei-iatsenko]

I was very close to do it by myself, so thanks for the handling that.

I have a few questions though:

1. Why "threads" but not "priscina"? https://npm-compare.com/piscina,threads,workerpool it seems priscina much more popular

Also could you elaborate a bit about most notable changes, considerations, difficulties. to have more context for review.

From my side, i believe it's possible to implement for custom extractor as well, i will spent some time to tackle this.

[Zxilly]

1. Since I didn't know priscina, I initially wrote it directly using the native API of worker_thread. Then I realized that I needed a wrapper library, and after a quick search, I found threads.js, which seemed to meet my requirements.
2. I don't think there's anything special to note here. Just offload a function to a worker, design a serialization protocol that allows objects to be passed between threads, and that's all. The main issue is that we need to ensure that the results of multi-threaded execution are consistent with single-threaded execution, so I added a lot of extra sorting to get deterministic results.
3. I have no idea about custom extractors. How should the function be passed between threads? Perhaps toString and then eval, but if the extractor contains any additional imports, this won't work.

[timofei-iatsenko]

@Zxilly i'll take it from here, thanks for contribution.

[Zxilly]

I will also continue working on this patch. Perhaps we can standardize the way workers are handled? Or add more unit tests? I'll see what I can do.

[timofei-iatsenko]

I closed all other PR's and reopened this one. Here is my comment from other PR;

1. I reverted multithreading changes for experemental extractor. The splitting work for worker pool was done in in the incorrect place. So instead of parallelezing work between workers, it actually create many parallel pools, with one worker only. This need more effort to make it work correctly, i would like to focus more on the stable extractor / compile first and then optimize new extractor.
2. I added support for custom extractors as well, instead of passing a config as a parameter to a worker, i'm re-creating the config in every of them. I also updated jiti (a lib used to read esm/ts/js configs on the fly in nodejs) version to make sure that it using a cache for config transpilation, also i'm passing a resolved config path to a worker, to avoid extra config crawling.
3. I simplified the code, and made it more aligned to the non-multithreading version. Instead of passing file content to the worker, worker reading fs by itself.

I'm also going to review the compile approach, probably some changes would be done there as well. I'll keep you posted.

[codecov]

Codecov Report

❌ Patch coverage is 74.70356% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.17%. Comparing base (6bb8983) to head (60c03ab).
⚠️ Report is 217 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/cli/src/workers/compileWorker.ts	0.00%	11 Missing ⚠️
packages/cli/src/workers/extractWorker.ts	0.00%	8 Missing ⚠️
.../src/extract-experimental/workers/extractWorker.ts	0.00%	7 Missing ⚠️
packages/cli/src/lingui-compile.ts	74.07%	3 Missing and 4 partials ⚠️
packages/cli/src/api/catalog/extractFromFiles.ts	78.57%	2 Missing and 4 partials ⚠️
packages/cli/src/api/workerLogger.ts	0.00%	5 Missing ⚠️
.../extract-experimental/extractFromBundleAndWrite.ts	73.68%	4 Missing and 1 partial ⚠️
packages/cli/src/api/compile/compileLocale.ts	89.18%	1 Missing and 3 partials ⚠️
packages/cli/src/lingui-extract-experimental.ts	88.88%	2 Missing and 1 partial ⚠️
packages/cli/src/lingui-extract.ts	83.33%	2 Missing and 1 partial ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2313      +/-   ##
==========================================
- Coverage   77.05%   76.17%   -0.88%     
==========================================
  Files          84       99      +15     
  Lines        2157     2636     +479     
  Branches      555      685     +130     
==========================================
+ Hits         1662     2008     +346     
- Misses        382      504     +122     
- Partials      113      124      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[timofei-iatsenko]

Regarding the coverage - coverage is not gathered for worker files, it seems they are not get instrumented since they are spawned in a separate thread.

The other files just extracted from existing places, so coverage level was not changed for them.

[Zxilly]

https://github.com/bcoe/c8 This may be helpful for coverage collection.

[timofei-iatsenko]

#2320