JSDoc comments syntax highlighting does not match parsed state of @type / @typedef blocks

[myfonj]

Type: Bug

JS/TS language service is quite permissive for JSDoc comment block structure and does not have problems with line breaks and optional leading asterisks (good), but sadly the JavaScript.tmLanguage definitions are more strict and fall back to regular comment block in any sign of irregularity in the structure of the comment. For example simple line break before closing */ of "line" JSDoc comment makes it appear like a normal insignificant comment:

/** @type {string} */
var a = "OK, highlighted as JSDoc.";
/** @type {string}
*/
var b = "ERROR, highlighted like a regular comment.";

See this this abomination of a JSDoc @typedef that is still correctly processed (despite inconsistent usage of leading asterisks and line breaks) yet not highlighted as such. Notice errors on the bottom and the tooltip that shows that all JSDoc blocks are processed:

// @ts-check

/** @typedef
    * {
    typeof
    * ABC [
         number
    * ]
    }
           *       Ch -
   this
 - still
 * works
*/
// ↑ ERROR, highlighted as a comment, but processed

const ABC = /** @type {const} */(["a", "b", "c"]);
	
/** @type {Ch}
*/
// ↑ ERROR, highlighted as a comment, but processed
var x = 'd';

/** @type {Ch} */
// ↑ OK, highlighted
var y = 'd';

---

On a personal note, I still do not fully understand why is highlighting done separately from language services. It feels that two parts are doing same work, parsing and tokenizing, but one does it for real and the other just fakes it. Is the Text TextMate grammar processing significantly faster or what is the reason that allows issues such as this one to happen?

---

VS Code version: Code - Insiders 1.88.0-insider (15a0f3c, 2024-02-29T05:47:41.883Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz (8 x 2112)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	15.76GB (2.26GB free)
Process Argv
Screen Reader	no
VM	0%

Extensions (1)

Extension	Author (truncated)	Version
errorlens	use	3.16.0

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
vsaa593cf:30376535
py29gd2263:30784851
c4g48928:30535728
2i9eh265:30646982
962ge761:30841072
pythongtdpath:30726887
welcomedialog:30812478
pythonidxpt:30768918
pythonnoceb:30776497
asynctok:30898717
dsvsc013:30777762
dsvsc014:30777825
dsvsc015:30821418
pythontestfixt:30866404
pythonregdiag2:30926734
pyreplss1:30879911
pythonmypyd1:30859725
pythoncet0:30859736
pythontbext0:30879054
accentitlementst:30870582
dsvsc016:30879898
dsvsc017:30880771
dsvsc018:30880772
bf62j303:30968145
pydisww2:30959801
cppperfcontrol:30963551
d34g3935:30961436
edj9j583:30969294
fegfb526:30952798
e3gdj431:30946825
ccp1r3:30958150
pythonait:30973460

[RedCMD]

TextMate runs on the render thread
which must stay fast
any lag will cause cursor freezing etc

for performance reasons, TextMate only parses one line at a time
making it rather difficult to do a context aware multiline tokenizing

however it is possible to make the grammar more lenient
allowing it to follow the LSP more closely

[myfonj]

however it is possible to make the grammar more lenient

I see. So some digging revealed among swarm of other JSDoc related bugs microsoft/TypeScript-TmLanguage#467 (closed at this moment), what seems like the most fitting place to continue.

---

OT/cont

TextMate runs on the render thread
which must stay fast
any lag will cause cursor freezing etc

Understood. But still sounds a bit scary: IIUC the "cursor freezing" caveat, the syntax highlighter in the render thread is sorta-kinda synchronous and blocking process, right? What basically means it has even higher priority than keyboard input? If so, it sounds like an odd design choice for software what's primary purpose is to … edit text files by typing or similar user input.
(And yes, I have experienced keyboard lag in VSC, but always thought it is result of certain extension/file combination (humongous markdown with several MD-related extensions, for example) gobbing CPU so much that OS itself runs into problems with HW signals/interrupts, or something, not that VSC waits for information what colour should the last character have.)