feat: serialize AST node back to wikitext string by linonetwo · Pull Request #8258 · TiddlyWiki/TiddlyWiki5

linonetwo · 2024-06-12T17:28:01Z

Currently just a demo, I will gradually move code from https://github.com/tiddly-gittly/wikiast/tree/master/packages/wikiast-util-to-wikitext to here.

Try it with tiddler (Well, if official website have share plugin installed, I can put a link here):

AAA test

---

\```js
var match = reEnd.exec(this.parser.source)
\```

end

and run this in console:

$tw.utils.serializeParseTree($tw.wiki.parseTiddler('New Tiddler').tree)

vercel · 2024-06-12T17:28:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Jun 12, 2024 5:39pm

… text" This reverts commit 8a12498.

pmario · 2024-06-13T08:24:26Z

The problem here is, that the output of you test does not produce the same rendered HTML output as the original source. See the screenshot.

linonetwo · 2024-06-13T09:27:44Z

First line is wrapped by a p element, but I haven't find the rule that produce the p element. And there was no p html in the source, so I may need to add some attribute to ast to show this p element is "fake".

Anyway, this is just a demo for API, do you like the API name and their usage? The further task is just adding exports.serailize = xxx to each rule.

pmario · 2024-06-13T09:33:59Z

First line is wrapped by a p element, but I haven't find the rule that produce the p element. And there was no p html in the source, so I may need to add some attribute to ast to show this p element is "fake".

IMO the current parser has a "systemic" problem with redundant P-tags, that we probably may need to solve first.

Working around this problem with "fake" elements will just increase your code complexity, instead of solving the underlaying problem. It will also create an other dependency, that will make solving the problem, in the core, in backwards compatible way even harder as it is now.

linonetwo · 2024-06-13T14:04:01Z

redundant P-tags

Can't remove then, many of my plugin style sheet is already depending on this structure. And p > text for block is following HTML5 semantic.

Another option is handle it like text node, regard it as a special case.

pmario · 2024-06-13T14:05:53Z

And p > text for block is following HTML5 semantic.

That's true, but we do have a lot of p > div which should not happen at all.

linonetwo · 2024-07-29T16:47:05Z

Handle the block rule by adding a rule: 'parseBlock' to WikiParser.prototype.parseBlock.

I've added some test for it.

…n of it

github-actions · 2025-10-23T16:55:39Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2423.3 KB
PR	2426.2 KB

Diff: ⬆️ Increase: +2.9 KB

Jermolene

Thank you @linonetwo for your dedicated work on this PR for so long. These are very useful changes, and the implementation is exemplary. I appreciate that you addressed my feedback several months ago, and it is unfortunate that there has been such long wait before we could commit changes for v5.4.0.

There’s a minor point below, and then once the feedback of @saqimtiaz has been addressed I’ll be able to merge this PR.

Co-authored-by: Jeremy Ruston <jeremy@jermolene.com>

github-actions · 2025-10-29T18:49:29Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2423.3 KB
PR	2426.2 KB

Diff: ⬆️ Increase: +2.9 KB

github-actions · 2025-10-29T19:01:21Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2423.3 KB
PR	2426.2 KB

Diff: ⬆️ Increase: +2.9 KB

saqimtiaz

@linonetwo this looks good, thank you for all of your work on this and for your patience.

@Jermolene this looks good from my perspective.

Jermolene · 2025-10-29T21:21:57Z

Great stuff, thank you @linonetwo.

linonetwo · 2025-10-30T08:03:38Z

Cool, thanks for review, being careful is right.

* refactor: extract a new $tw.wiki.getParser * feat: allow $tw.utils.getParseTreeText to render other rules' text * feat: two example getText handler * Revert "feat: allow $tw.utils.getParseTreeText to render other rules' text" This reverts commit 8a12498. * refactor: keep original getParseTreeText not touched * refactor: use serialize in rules * refactor: $tw.utils.extend({},options) -> options || {} * Update codeinline.js * Create test-wikitext-serialize.js * DEBUG: only run my tests for development, remove before PR merge * lint: if * feat: add rule: 'parseBlock' metadata * feat: handle tailing \n that may be missing * feat: allow recursive * feat: generate more rule and tests * feat: generate more rule and tests * fix: remove pragma:true, otherwise following text will become children of it * fix: condition manually Deekseek is silly * fix: some test * fix: some test * feat: $tw.utils.serializeAttribute * fix: use "" for string param * feat: list * refactor: ' -> " * fix: parsemode don't have node * fix: render invisible comment and parsemode as data element * feat: add void: true, in ast node to prevent render * feat: use void widget, so methods always return a widget * feat: ast to use new widget type void * test: add rule: 'parseBlock' and isRuleEnd: true * lint: quote * Update widget.js * fix: void node need to handle its children * Update test-wikitext-parser.js * lint: quote * Update void.js * Update test-wikitext-parser.js * fix: macrodef with comment (void node) not working * lint: ' -> " * feat: add to styleblock * feat: styleblock * feat: styleinline * Update table.js * lint: useless comments * feat: transcludeblock * refactor: reuse block on inline when possible * feat: use void node to carry important info for typedblock * feat: run all tests * lint: useless ai generated comments * Update conditional.js to not include space * Update test-wikitext-serialize.js * Update conditional.js * refactor: move tiddlers to /data * refactor: no need for new $tw.Wiki() * lint: double quote * refactor: lowercase the parseblock rule name * fix: Wiki parser initialize blockRuleClasses only when first new an instance * feat: restore inline macro def * fix: macro in widget param * fix: positional attribute in macro call * fix: table space and horizrule block new line * feat: make sure block rule all have \n\n for visiblity * lint: function param * fix: empty list item * feat: add \n\n based on isBlock, if could also be inline * fix: conditional without elseif * refactor: use isBlock in macrodef to know inline or block * fix: link may not have attribute and children * DEBUG: render result and diff below body only on browser DEBUG: render result below body only on browser DEBUG: render result below body DEBUG: fix build DEBUG: show render result as ViewTemplate * fix: remove pad space in /> * test: remove pad space in /> * Revert DEBUG: render result and diff below body only on browser * refactor: fold commentText variable * refactor: fold long comment * fix: double quotes for parameter values * Update void.js * refactor: move all exports.serialize = function(tree,serialize) { to plugin * fix: expost listTypes from core, and require it in plugin * refactor: move serializeWikitextParseTree to plugin and init it * refactor: move serializeAttribute util also to the plugin * fix: Delete unused file * Update macrodef.js * Update test-wikitext-parser.js * lint: fix * Update plugins/tiddlywiki/wikitext-serialize/rules/filteredtranscludeblock.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update core/modules/widgets/void.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/tiddlywiki/wikitext-serialize/rules/list.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/tiddlywiki/wikitext-serialize/rules/list.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/tiddlywiki/wikitext-serialize/rules/styleblock.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove unused methods from VoidNodeWidget Deleted render, execute, and refresh methods from VoidNodeWidget as they are no longer needed. The widget now only inherits from the base Widget class and exports the constructor. * docs: about regex in styleinline.js * Update parsetree.js * Update core/modules/widgets/void.js Co-authored-by: Jeremy Ruston <jeremy@jermolene.com> * feat: Ensure at least one space after the style/class --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jeremy Ruston <jeremy@jermolene.com>

flibbles · 2026-02-28T23:36:02Z

Should this be merged into core? Separated as it is, it won't stay synchronized. It's already appears to be out of date, since it doesn't have a mvvdisplayinline rule. If the serialize rules are right there inside the wikirule modules, then this won't be a problem going forward.

Also, plugins like Relink can't take advantage of this unless end-users include this plugin too.

Edit: Actually, nvm. This won't work for Relink anyway, so I've no need for it to be in core.

pmario · 2026-03-01T02:39:25Z

I think, this PR started before MVVs have been a thing. There are even more elements where they have not been fully implemented. From my point of view MVV development is not finished yet.

There has been an experimental PR that was rejected. It contained more MVV related changes. The process is not finished yet.

linonetwo · 2026-03-02T12:12:28Z

Anyone could just at me when you have trouble implementing the serialization rule. Me and my AI slaves are always willing to help, as long as I'm still using TiddyWiki.

I understand you may forget or don't want to do this unnecessary extra step.
But as WYSIWYG editor start to provide visual editing experience in 5.5.0 or 5.6.0, It will be more useful.

linonetwo added 3 commits June 13, 2024 01:23

refactor: extract a new $tw.wiki.getParser

37338b1

feat: allow $tw.utils.getParseTreeText to render other rules' text

8a12498

feat: two example getText handler

09c2e71

linonetwo added 2 commits June 13, 2024 01:29

Revert "feat: allow $tw.utils.getParseTreeText to render other rules'…

a0708a1

… text" This reverts commit 8a12498.

refactor: keep original getParseTreeText not touched

f882cbd

vercel Bot deployed to Preview June 12, 2024 17:31 View deployment

refactor: use serialize in rules

be3f037

vercel Bot deployed to Preview June 12, 2024 17:39 View deployment

linonetwo changed the title ~~Feat/to string~~ feat: serialize AST node back to wikitext string Jun 12, 2024

pmario reviewed Jun 13, 2024

View reviewed changes

Comment thread core/modules/wiki.js Outdated

linonetwo added 5 commits July 29, 2024 21:43

Merge remote-tracking branch 'upstream/master' into feat/to-string

4ddcbc6

refactor: $tw.utils.extend({},options) -> options || {}

4e896d1

Update codeinline.js

179651a

Create test-wikitext-serialize.js

8cbf855

DEBUG: only run my tests for development, remove before PR merge

a075303

linonetwo mentioned this pull request Jul 29, 2024

Add start and end properties to WikiText AST nodes for all elements. #7866

Closed

linonetwo added 3 commits July 30, 2024 00:43

lint: if

05dec72

feat: add rule: 'parseBlock' metadata

1a6a071

feat: handle tailing \n that may be missing

84e27c0

linonetwo added 4 commits July 30, 2024 00:56

feat: allow recursive

078967c

feat: generate more rule and tests

2aff9fe

feat: generate more rule and tests

bfd116a

fix: remove pragma:true, otherwise following text will become childre…

d067871

…n of it

Update parsetree.js

756d657

Jermolene reviewed Oct 28, 2025

View reviewed changes

Comment thread core/modules/widgets/void.js Outdated

Update core/modules/widgets/void.js

8cbd6c2

Co-authored-by: Jeremy Ruston <jeremy@jermolene.com>

feat: Ensure at least one space after the style/class

423973c

linonetwo requested a review from saqimtiaz October 29, 2025 19:06

saqimtiaz approved these changes Oct 29, 2025

View reviewed changes

Jermolene merged commit 20d6be1 into TiddlyWiki:master Oct 29, 2025
11 checks passed

github-project-automation Bot moved this from Ready to Done in Planning for v5.4.0 Oct 29, 2025

linonetwo mentioned this pull request Oct 30, 2025

Feat: allow link to ^id anchor #7744

Open

yaisog mentioned this pull request Nov 9, 2025

Allow recursive \import #9389

Closed

5 tasks

Jermolene mentioned this pull request Nov 9, 2025

Summary of PRs that need TW changenotes #9406

Open

55 tasks

Copilot AI mentioned this pull request Nov 9, 2025

Make up for my missing change note #9409

Merged

Copilot AI mentioned this pull request Nov 10, 2025

Fix/validate release note type #9414

Closed

Jermolene mentioned this pull request Nov 10, 2025

Implement Markdown to wikitext conversion #9415

Open

yaisog added a commit to yaisog/TiddlyWiki5 that referenced this pull request Nov 11, 2025

Incorporate changes from PR TiddlyWiki#8258

d010584

linonetwo mentioned this pull request Nov 13, 2025

Fix/serialize close html tag #9437

Merged

github-actions Bot mentioned this pull request Nov 25, 2025

Button that converts markdown to wikitext #9462

Open

pmario mentioned this pull request Feb 25, 2026

[IDEA] Implement some Markdown syntax in the wikitext parser #9327

Open

linonetwo mentioned this pull request Mar 10, 2026

feat: filter serialize, and mvvdisplayinline serializer #9722

Open

pmario mentioned this pull request Jul 2, 2026

[Report] Plugin wikitext-serialize corrupts inline code that contains a backtick #9898

Open

pmario mentioned this pull request Jul 12, 2026

Wrap hardlinebreaks regions in a void node #9924

Draft

Uh oh!

Conversation

linonetwo commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel Bot commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmario commented Jun 13, 2024

Uh oh!

linonetwo commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmario commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

linonetwo commented Jun 13, 2024

Uh oh!

pmario commented Jun 13, 2024

Uh oh!

linonetwo commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2025

📊 Build Size Comparison: empty.html

Uh oh!

Jermolene left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2025

📊 Build Size Comparison: empty.html

Uh oh!

github-actions Bot commented Oct 29, 2025

📊 Build Size Comparison: empty.html

Uh oh!

saqimtiaz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jermolene commented Oct 29, 2025

Uh oh!

linonetwo commented Oct 30, 2025

Uh oh!

flibbles commented Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmario commented Mar 1, 2026

Uh oh!

linonetwo commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

linonetwo commented Jun 12, 2024 •

edited

Loading

vercel Bot commented Jun 12, 2024 •

edited

Loading

linonetwo commented Jun 13, 2024 •

edited

Loading

pmario commented Jun 13, 2024 •

edited

Loading

linonetwo commented Jul 29, 2024 •

edited

Loading

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

saqimtiaz left a comment •

edited

Loading

flibbles commented Feb 28, 2026 •

edited

Loading

linonetwo commented Mar 2, 2026 •

edited

Loading