[IDEA] Serailize/toString of AST node

[linonetwo]

Is your feature request related to a problem? Please describe.

Need to get original wikitext from AST, when writing WYSIWYG editor.

Describe the solution you'd like

Add rule = exports.name field to ast node, and add exports.toString to wikiparsers.

In toString, try call handleChildrenToString. (The problem is where to find this method, we can't access $tw in the wikiparser. Can we put it on parser passed from exports.init = function(parser) {?

Describe alternatives you've considered

Now it is maintained in https://github.com/tiddly-gittly/wikiast, but is quickly outdated.

Additional context

I remember some other issue we talk about adding a rule field to the ast.

[saqimtiaz]

Add rule = exports.name field to ast node

See e4c682d where this has been implemented.

Part of the problem that still remains is that the parse tree does not have enough information about whitespace, comments and pragma so recreating the original wikitext still isn't possible.

[pmario]

Need to get original wikitext from AST, when writing WYSIWYG editor.

That's not really possible at the moment. While we did add a lot of new start and end positions to the AST lately, the <!-- comments --> are not there. So at the moment it's kind of impossible to create lossless wikitext from the TW AST.

I also think there is not enough information about function, procedure calls in the AST. eg: <<test>> and <$macrocall $name=test may not be distinguishable. -- But I'm not really sure about that one. -- But "The devil is in the details" atm

[linonetwo]

There is an unused method getParseTreeText, it used to work for text-slicer plugin, but it's useless now.

Old usage (don't exist anymore): https://github.com/eyecatchup/TiddlyWiki5/blob/72ed4b26731dd3bbac948af6965b050f33ed1c1c/plugins/tiddlywiki/text-slicer/slicer.js#L79C23-L79C39

https://github.com/Jermolene/TiddlyWiki5/blob/4bda8cfee6568fcdf2fd573c5303dbbd5751abd3/core/modules/utils/parsetree.js#L106-L124

Usage: $tw.utils.getParseTreeText($tw.wiki.parseTiddler('HelloThere').tree)

I can gradually add wiki rule support to it. Not need to be lostless, it only need to handle 80% of everyday usecase. People who use WYSIWYG editor won't write comment or procedure, support for then can be delayed.

See e4c682d where this has been implemented.

https://github.com/Jermolene/TiddlyWiki5/blob/d3e62ec56a91b004bd74b77273209e58ffb81f05/core/modules/parsers/wikiparser/wikiparser.js#L219

I see. This is not very obvious. I was only reading files in core\modules\parsers\wikiparser\rules

[Jermolene]

Thanks @linonetwo I hadn't noticed that getParseTreeText() is no longer used.

I can gradually add wiki rule support to it. Not need to be lostless, it only need to handle 80% of everyday usecase. People who use WYSIWYG editor won't write comment or procedure, support for then can be delayed.

getParseTreeText() is intended to just extract the value of text nodes. I think you're proposing a function to convert a parse tree back to wiki text? I would think that that should be a separate function.

I take your point that it wouldn't be necessary to handle all wikitext constructions. It would be interesting to see how complex the implementation would be, but if it's not excessive then I would be happy to see that go into the core.

[pmario]

I take your point that it wouldn't be necessary to handle all wikitext constructions. It would be interesting to see how complex the implementation would be, but if it's not excessive then I would be happy to see that go into the core.

I think @flibbles knows best, what's needed for a "rewrite" of tiddler content. IMO the relink-plugin does exactly that.

@linonetwo -- I would have a closer look at the relink-AST (if there is one) and I would try to get or reuse the API from the relink plugin. Most users who want to use WYSIWYG will need the relink-plugin anyway.

[linonetwo]

Yes, this is interesting, how relink works when there was no start/end pos in the AST?

After reading for a while, I find it is simply use regex to replace old value, not using AST

https://github.com/flibbles/tw5-relink/blob/4e3805ec149817cbb8e1ea7315cb8b4b72b1d25b/plugins/relink/js/utils/substitution.js#L33

[pmario]

... I find it is simply use regex to replace old value, not using AST

That may be true. @flibbles -- should know exactly.

[flibbles]

Yes, this is interesting, how relink works when there was no start/end pos in the AST?

After reading for a while, I find it is simply use regex to replace old value, not using AST

You're looking at the method used to handle substitution attributes. Relink doesn't just use regexp. It also doesn't utilize the existing AST since it's incomplete for reproducing the original text. It modifies the Wikitext parser and has it "parse" text, with each wikitext rule module getting new report and relink methods. the "AST" it produces is the modified text.

[linonetwo]

@Jermolene Need to add a new parse rule type

exports.name = "commentblock";
exports.types = {void:true, block:true, pragma:true};

When adding void:true, the AST node will not be rendered.

Currently without this, commentblock / parsermode will render to some text, cause style error, because they break the stylesheet tiddlers.

[Jermolene]

@Jermolene Need to add a new parse rule type

Yes, that sounds like a reasonable solution.

[linonetwo]

If you are interested, feel free to PR more test to #8258 . I've already pass all basic tests.

In some case, original whitespace is not preserved, may add/remove some of them to make result beautiful, like prettier. And user don't care about this, because in WYSIWYG editor, user don't care underlying text, they just want a text file so it feels safe.