Transfrom languages to use the base property

[DmitrySharabin]

---

This is part 2 of 2 in a stack made with GitButler:

•  2  Transfrom languages to use the base property #3991 👈
•  1  Introduce the concept of base language from the simplify branch #3990

[github-actions]

No JS Changes

Generated by 🚫 dangerJS against eb97fb9

[LeaVerou]

Won't this delete these from the actual base language, rather than this one?

[LeaVerou]

Also, can we somehow tell TS that if the language has a base property, then base is not null so we don't have to use base! everywhere?

[DmitrySharabin]

Won't this delete these from the actual base language, rather than this one?

Nope, we pass a clone of the base language as the value of base. See line 173 in #3990 (the base PR):

const baseGrammar = base && cloneGrammar(required(base.id), base.id);

[DmitrySharabin]

Also, can we somehow tell TS that if the language has a base property, then base is not null so we don't have to use base! everywhere?

It bothers me as well. I'll take a closer look.

[LeaVerou]

Thanks. Let's see what your TS research brings up.
Some other questions:

• Are there any languages where this type of rewrite produced worse code?
• Are there any languages that you couldn't transform to use base even though conceptually they do use a base language?

[DmitrySharabin]

Let's see what your TS research brings up.

Okay, I seem to have found a way to distinguish (on the level of TS) between cases where a language definition has a base property and those where it doesn't. 🎉 See #3990.

Are there any languages where this type of rewrite produced worse code?

Can't think of any. But there are a couple of languages where the grammar function returns an empty object (e.g., cilkc, cilkcpp, etc.), which looks a bit weird. However, before, those languages extended the base language with nothing (an empty object). So it's just the same bizarre thing, but in a different place. 😅

[DmitrySharabin]

Are there any languages that you couldn't transform to use base even though conceptually they do use a base language?

There are some. For example, coffeescript. The transformation is rather straightforward, but one group of tests (inline-javascript_feature) still fails. And I don't understand (yet) why. 🤷‍♂️

In cpp, we use the language we are (partially) defined (by extending the base language and modifying it on the way):

prism/src/languages/cpp.ts

Line 109 in 31b467f

const baseInside = { ...cpp };

I don't know how to address that case either.

In tt2, the language has only one rule—tt2 but inside, it's the extended version of the base language. Techicaly, we shouldn't extend the value returned from the grammar() function of this language and use it as-is. I'd say, this language definition is an edge case for the concept of a base language.

[DmitrySharabin]

For example, coffeescript. The transformation is rather straightforward, but one group of tests (inline-javascript_feature) still fails. And I don't understand (yet) why. 🤷‍♂️

Some observations. The coffescript grammar defines an inline-javascript token. I compared how this token looks in the resolved grammar in the source and transformed languages. And it turned out that now (not transformed), coffeescript['inline-javascript'].script.inside looks exactly the same as it's defined, i.e.,
inside: 'javascript'.

However, after I transformed the language to use the base property, the same part of the resolved grammar looks different. Its inside property's value is an object—(resolved) grammar of the javascript language, and, for some reason, it acts differently when it is being tested. It seems to me that's the reason why the inline-javascript_feature fails.

[LeaVerou]

Let's see what your TS research brings up.

Okay, I seem to have found a way to distinguish (on the level of TS) between cases where a language definition has a base property and those where it doesn't. 🎉 See #3990.

Yay! Though this complexity makes me wonder once more whether we should consider switching back to JS + JSDoc. We keep spending time jumping through TS hoops. I’m also worried how the TS rewrite will affect performance, since it's a fairly significant abstraction.
Do you have a sense of how much work it would take? I wonder if in the long term it could save us time. 🤔

Are there any languages where this type of rewrite produced worse code?

Can't think of any. But there are a couple of languages where the grammar function returns an empty object (e.g., cilkc, cilkcpp, etc.), which looks a bit weird. However, before, those languages extended the base language with nothing (an empty object). So it's just the same bizarre thing, but in a different place. 😅

They shouldn't have to. Returning undefined (what happens when there is no return) should just work.

[LeaVerou]

For example, coffeescript. The transformation is rather straightforward, but one group of tests (inline-javascript_feature) still fails. And I don't understand (yet) why. 🤷‍♂️

Some observations. The coffescript grammar defines an inline-javascript token. I compared how this token looks in the resolved grammar in the source and transformed languages. And it turned out that now (not transformed), coffeescript['inline-javascript'].script.inside looks exactly the same as it's defined, i.e., inside: 'javascript'.

However, after I transformed the language to use the base property, the same part of the resolved grammar looks different. Its inside property's value is an object—(resolved) grammar of the javascript language, and, for some reason, it acts differently when it is being tested. It seems to me that's the reason why the inline-javascript_feature fails.

I wonder if we end up accidentally modifying an existing object when we think we're working with a clone.

[DmitrySharabin]

For example, coffeescript. The transformation is rather straightforward, but one group of tests (inline-javascript_feature) still fails. And I don't understand (yet) why. 🤷‍♂️

Some observations. The coffescript grammar defines an inline-javascript token. I compared how this token looks in the resolved grammar in the source and transformed languages. And it turned out that now (not transformed), coffeescript['inline-javascript'].script.inside looks exactly the same as it's defined, i.e., inside: 'javascript'.
However, after I transformed the language to use the base property, the same part of the resolved grammar looks different. Its inside property's value is an object—(resolved) grammar of the javascript language, and, for some reason, it acts differently when it is being tested. It seems to me that's the reason why the inline-javascript_feature fails.

I wonder if we end up accidentally modifying an existing object when we think we're working with a clone.

I also think we might. However, I can't pinpoint exactly where it happens—something to explore in the next iteration.

[netlify]

✅ Deploy Preview for dev-prismjs-com ready!

Name	Link
🔨 Latest commit	eb97fb9
🔍 Latest deploy log	https://app.netlify.com/projects/dev-prismjs-com/deploys/6883a3bb1a5b200008d5694d
😎 Deploy Preview	https://deploy-preview-3991--dev-prismjs-com.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.