Experimental - Wikitext serialize tests - for discussion#9922
Draft
pmario wants to merge 25 commits into
Draft
Conversation
Extend the wikitext-serialize fixtures with documented but uncommon syntax, mainly containment forms like triple quoted attributes. * Source the cases from the tw5.com docs; undocumented parser behaviour may still be missing. * Cover quote containment in attributes, macro parameters and parameter defaults, plus substituted values with embedded backticks. * Cover nested definitions with named \end, nested conditionals, nested block quotes and single line procedures. * Cover table pseudo rows and cell merging, description list nesting, quote lists, heading classes and ext link schemes. * Add a fixture for the 5.4.0 ((var)) display syntax. * Expect 16 of 51 fixtures to fail: they encode behaviour the serializer cannot reproduce yet.
Separate upstream fixtures from the new coverage so a failing spec points at a small single purpose file. * Restore the original X.tid fixtures to their upstream content; they form a control group that must stay green. * Move the docs-based cases from the last commit into X-docs.tid files. * Move the parser-parity cases into X-new.tid files. * Expect 19 of 77 specs to fail; all failures are in the new files.
Use parse tree source positions to reproduce the original formatting of attributes and macro parameters, and add normalization tests. * Thread the serializer options through rule serializers and serializeAttribute; the test driver now passes the tiddler source. * Detect the original quote style (single, double, triple, bracket, backtick, unquoted, valueless) from the source via node positions and reuse it when it still fits the value; fall back to content-driven minimal quoting instead of always double quotes. * Serialize ((var)) attribute references and keep the space before >> when a parameter ends with >. * Add a normalization test harness: compound tiddlers tagged wikitext-serialize-normalize-spec hold Input and ExpectedOutput subtiddlers, with an optional canonical mode without source access. * Add five X-normalized fixtures; failures drop from 19 to 13, the rest are structural rule gaps.
Make the \whitespace pragma round-trippable and reproduce definition pragmas verbatim from their source slice. * Return a void parse tree node from the core whitespace rule, exactly like \rules does; it was the only pragma leaving no trace in the tree, so serializers could never see it. Rendering is unchanged and the cost is within benchmark noise (0.1% on a PageTemplate render). * Add $tw.utils.serializeFromSource returning a node's exact source slice, guarded by fragments that must still appear in it, so edited trees fall back to canonical serialization. * Use the slice in the macrodef and fnprocdef serializers to preserve named \end markers, default quoting and single line definitions. * Add a whitespace serializer rule that recovers the pragma separator from the source gap; the canonical fallback emits a blank line. * Rename the normalize fixture subtiddler ExpectedOutput to ExpectedWikitext to avoid confusion with the rendered ExpectedResult of the wiki test specs. * Add fixtures: whitespace round-trip and spacing normalization, \end trailing spaces for \procedure, whitespace pragmas and indented nested definitions inside procedure bodies; the definition fixtures now round-trip verbatim.
Serialize the three inline rules that still lost formatting or content: html, image and the multi valued variable display. * Slice the opening tag from the source so multi line attribute layouts survive; a fragment check falls back to canonical serialization for mutated attributes. * Restore whitespace only gaps between children from the source; they cover line breaks eaten by \whitespace trim and the blank line that switches an element to block mode, and replace the fixed blank line joiner of block rules with the true gap before a close tag. * Omit the close tag when the parser recorded an implicit one. * Serialize all image attributes in order instead of only width and height, so class attributes are no longer dropped. * Add the mvvdisplayinline serializer: decompile the join filter the parser generates back into the ((var)) and (((filter))) forms. * Turn the attribute whitespace fixture canonical: the opening tag slice preserves the spacing in fidelity mode, and canonical html attributes quote their values.
Fix the conditional serializer losing else content and reproduce the original clause layout from the source. * Stamp isConditional on the list widgets synthesized by parseIfClause and record each elseif clause span, so consumers can tell synthesized clauses from real $list widgets without sniffing attributes. * Distinguish elseif clauses by that flag; a nested <%if%> or a $list widget at the start of an else body was misread as an elseif clause and mangled or dropped. * Stitch the serialized clause bodies together with the source text between them: the clause markers only exist in the source, and the stitch preserves marker spacing and clause indentation. Mutated trees fall back to a simplified canonical emitter.
Serialize inline code runs with the delimiter they were written with, or the minimal safe delimiter without source access. * Read the original delimiter from the source at the node position; the parse tree does not record whether ` or `` matched. * Fall back to double backticks when the text contains a backtick or is empty, eg ``a`b`` and ````; two bare backticks would reparse as an opening double delimiter. * Extend the fixtures with embedded, leading and empty backtick runs and pin the canonical minimal delimiter choice.
Stop inventing a blank line between a pragma area comment and the following pragma. * Recover the separator from the source gap between the comment and its first chained child; a comment directly above a \define kept a single newline in the source but serialized with a blank line. * Keep the blank line as the canonical fallback for body comments and trees without source access. * Extend the fixture to pin both separator styles: a comment directly above the pragma and one separated by a blank line.
ATTENTION: this commit contains a core parse tree change that needs its
own upstream issue and PR, separate from the serializer plugin PR. Move
core/modules/parsers/wikiparser/rules/hardlinebreaks.js and
editions/test/tiddlers/tests/test-wikitext-parser.js into that PR.
plugins/tiddlywiki/wikitext-serialize/rules/hardlinebreaks.js belongs
to the serializer PR but requires the core change to be merged first.
Give the hard linebreak region a container node so inner content keeps
its own rule names.
* Return a single void wrapper from the core rule instead of a flat
node array; the rule stamping overwrote inner rule names, e.g. bold
inside a region became rule hardlinebreaks, which lost the producer
for serialization and source mapping.
* Stamp the br nodes explicitly and retire the isRuleStart and
isRuleEnd flags; the wrapper carries the region span.
* Serialize the region as fences plus children; formatted content now
dispatches to its own rules, and the opening fence form ("""text or
a line break) is recovered from the source.
Needs UPSTREAM PR: $:/core/modules/parsers/wikiparser/rules/quoteblock.js Room for improvements: Extract the shared stitch helper (TiddlyWiki5-99v); fix the terminator newline swallow (TiddlyWiki5-40b); drop the serializeCanonical legacy fallbacks, core and plugin ship together. Record what the quote parser knows and reproduce quotes from the source instead of dropping citations, classes and nesting. * Record marker, userClasses and isQuoteCite in the core rule: the class attribute fuses user classes with the synthesized tc-quote, and without the marker depth a serializer cannot emit parseable nested quotes at all. * Serialize citations on the opening and closing lines, classes in dot form and nested quotes; cite children were dropped and every quote was labelled with the class text tc-quote. * Stitch the quote from the source in fidelity mode, preserving marker depth, indentation and citation spacing; strip exactly one blank line joiner so a real paragraph-final newline survives. * Extend the fixture with a five deep marker nesting case.
Needs UPSTREAM PR: $:/core/modules/parsers/wikiparser/rules/list.js Room for improvements: The row joiner gap logic is another cousin of TiddlyWiki5-99v. Record what the list parser knows and fix multi class items and blank line separators. * Record rowMarker and blankLineBefore on list item nodes: the parser walks the marker string and consumes the blank line between merged lists, then dropped both, forcing serializers to recompute markers from the tag nesting and read separators from the source. * Give every item class its own dot, eg *.first.second; multiple classes were emitted with a single dot and inner spaces. * Collect one row per item line; the blankLineBefore flag reproduces the separator without source access and the source gap refines it to the exact bytes.
Room for improvements: The leaf stitch is another cousin of TiddlyWiki5-99v; a mode canonical fixture for the grid reconstruction path is still missing. Serialize every documented table feature instead of plain grids only, and stop crashing on captions. * Stitch the table from the source between the inline leaf nodes of the cells and the caption: pipes, merge markers, alignment spaces and class rows only exist in the source. Containers are re-sorted by position because the parser moves the caption to the front. * Rebuild the grid canonically without source access: rowspan and colspan attributes are re-emitted as ~ and < marker cells, cell alignment as the space convention, valign as ^ and , prefixes, and captions, headers, footers and class rows as their pseudo rows. * Stop reading the caption container as a row list; its inline content crashed the old code on any |c row.
Avoid the collision with the established _canonical_uri field by calling the tree-only fallback path "ast-only" in the test harness and the plugin. * Switch the mode field to ast-only in the eight normalize fixtures and reword their descriptions * Rename Attribute-normalized-canonical.tid to Attribute-normalized-quotes.tid * Update the mode check and its comment in test-wikitext-serialize.js * Rename serializeCanonical to serializeFromTree * Reword the remaining canonical comments in parsetree.js and mvvdisplayinline.js
Deduplicate the source stitching so each rule shrinks to its tree-only emitter plus one helper call. TODO: Fold the list.js row joiner refinement onto recoverSourceGap; fix the quoteblock terminator newline swallow (TiddlyWiki5-40b). * Add $tw.utils.serializeStitched with options.children and options.isBoundary, stripping exactly one block joiner per gap * Add $tw.utils.recoverSourceGap for separators the tree does not record * Collapse the stitch copies in conditional.js, quoteblock.js and table.js * Stitch the html inner region instead of the per-gap restoreGap loop * Recover the pragma gaps in whitespace.js and commentblock.js with the helper * Drop the stray trailing newline after fnprocdef children
Prove the tree-only serializer guarantee across the whole fixture corpus and fix every defect the sweep surfaced. Needs UPSTREAM PR: $:/core/modules/parsers/wikiparser/rules/html.js Needs UPSTREAM PR: $:/core/modules/parsers/wikiparser/rules/quoteblock.js (amends the quoteblock annotation already flagged in aa2f0ea) * Add a third describe block that serializes every identity fixture without source, re-parses the output and deep-compares the trees; the comparison helper drops position fields and undefined keys and sorts orderedAttributes by name, so it does not depend on the assertion framework * Drop the final block joiner at the root, keeping the line end a trailing <<macro>> call or block widget needs * Quote macrodef and fnprocdef parameter defaults with the shared $tw.utils.quoteParameterDefault and emit ((var)) MVV defaults * Capture the quoteblock marker before a nested quote overwrites this.match, so an outer <<< no longer serializes as <<<< * Record blockPosition and blockContent on html nodes; isBlock fuses the parse position with the content mode and cannot be split again * Emit the html trailing joiner from blockPosition and the blank line after the open tag from blockContent * Update the parse tags expectations for the new html annotations * Document each describe block with a browser console recipe
ATTENTION: this commit is a standalone core bug fix that needs its own upstream issue and PR, separate from the serializer plugin PR. It contains only core parser rules and one new test file, so it can be cherry-picked as a unit. There is no dependency in either direction with the serializer plugin. A single newline before a closing marker was swallowed into the preceding paragraph text node, while the blank line form was not. * Anchor the quoteblock and styleblock terminators from the newline * Prefix the html and conditional block terminators with an optional newline; a mandatory anchor would break legal mid-line close tags * Leave the inline paths unchanged: a newline in an inline run is real content * Correct the (:?^| typo to (?:^| in the fnprocdef end marker regexp * Add test-wikitext-block-terminators.js, red without the rule fixes
Keep block conditional bodies block when serializing from the tree alone: the blank line after a clause marker was not recorded anywhere, so tree-only output re-parsed to inline clauses. Needs UPSTREAM PR: $:/core/modules/parsers/wikiparser/rules/conditional.js (extends the conditional annotations from d905319) * Stamp blockContent on the clause body containers in the core rule, mirroring the html annotation * Emit the blank line after each clause marker from blockContent in serializeFromTree * Add Conditional-new.tid covering block bodies closed by a single newline; its AST-only sweep spec was red before the annotation * Extend Html-new.tid with block content ending at the close tag
Give every core annotation unit its own red test, so each planned upstream PR ships with a spec that fails without its rule change. TODO: Move each spec file into its upstream PR per the extraction recipes in beads-planning plans/upstream-*-issue-pr.md. * Add test-wikitext-whitespace-pragma.js pinning the void marker node * Add test-wikitext-conditional-annotations.js pinning isConditional, the elseif clause span and blockContent * Add test-wikitext-quoteblock-annotations.js pinning marker, userClasses and isQuoteCite, plus the nested re-entrancy case * Add test-wikitext-list-annotations.js pinning rowMarker and blankLineBefore
Feed all 237 Output subtiddlers of the wiki-based rendering tests through the fidelity round-trip, then fix every gap the corpus surfaced. Needs UPSTREAM PR: $:/core/modules/parsers/wikiparser/rules/html.js (records openTagStart/openTagEnd for self-closing tags too; belongs to the html annotations unit) * Add a describe block round-tripping every Output subtiddler * Stitch the root blocks and paragraph children with the true source gaps, advancing over pragma chains by their deepest end * Emit source slices for text nodes whose span trim-equals their text, restoring the bytes eaten by \whitespace trim * Recover the pragma separators in macrodef, fnprocdef and import and stitch chained children with the new serializeChildren helper * Keep block emissions span exact: slices for transcludeblock, filteredtranscludeblock, codeblock and styleblock, options pass-through for their inline twins, tail repairs for horizrule and macrocallblock, no tail strip on stitched tables * Emit the suppression prefix of suppressed syslinks; the old check looked for children the text node never has * Slice macro calls and macro-valued attributes to preserve the author's parameter layout; MacroCallBlock-normalized becomes a mode: ast-only pair
ATTENTION: this commit is a standalone core change that belongs to the span discipline cluster (own upstream issue and PR, after the block terminator PR whose story it extends). It contains only core parser files and one new test file, so it can be cherry-picked as a unit. Block rules must consume their terminating line end to parse, but the stamped node spans swallowed it, so the separator byte belonged to a different owner per rule: transcludeblock, filteredtranscludeblock, horizrule, styleblock, typedblock and table included it, the other block rules and all pragmas did not. typedblock's span also started at its content instead of the opening $$$ marker. * Trim one trailing line end run from the last returned node at the block dispatch, never trimming into a child's span, e.g. an unterminated quote whose last text node keeps its newline * Trim the row and container ends in the table rule; the dispatch cannot reach them below the child span floor * Start the typedblock span at the opening $$$ marker * Add test-wikitext-block-spans.js, one spec per block and pragma rule asserting both span ends; red for seven spans without the fixes * Update the horizrule and table expectations for the shifted ends
ATTENTION: this commit is a standalone core change that belongs to the span discipline cluster and travels in the same upstream PR as the span commit before it. The parse position is not recoverable from the tree: macrocallblock produces the same node shape as its inline twin, but block position decides the separators around a node. The html rule already recorded it for itself; every block rule needs it before a serializer can own block separation centrally. Convention: blockPosition true on block dispatched nodes and on block positioned html elements, false on inline html elements (their isBlock fuses in the content mode, so absence must not mean inline there), absent on other inline nodes; paragraphs signal block position through their parseblock rule name. * Stamp blockPosition next to the rule name at the block dispatch * Assert the stamp per block rule in test-wikitext-block-spans.js * Add the field to the affected literal expectations: macrocall and filtered transclusion roots, both hr nodes, the table root and the block dispatched h1 inside a div
Make the serializer mirror the parser's span discipline: rule serializers emit exactly their own syntax, and the walker alone separates block siblings, with the true source gap in fidelity mode and one blank line from the tree alone. * Add isBlockNode reading blockPosition, with a legacy fallback to isBlock and the block rule name suffix for unannotated trees * Emit the separator between block siblings in the walker's array branch; nothing follows the last block * Delete the per rule blank line tails from heading, horizrule, codeblock, typedblock, table, quoteblock, list, styleblock, transcludeblock, filteredtranscludeblock, macrocallblock, html and conditional, and the tail repairs that inv 1 made dead code * Drop the joiner strip from serializeStitched and the root tail strip; no artificial joiners are left to undo * Serialize block children as one array in styleblock and quoteblock so the walker separates them * Give conditional markers and the html close tag their line end after block content, or a line terminated child like a heading would swallow them * Reduce the inline twins to plain delegation; their trims are no-ops now
Give each of the 89 serializer fixtures a description field, and each of the ten normalize pairs an info subtiddler that renders as documentation when pasted into a wiki. * Add a one line bottom line up front description to every fixture * Add an info subtiddler to each normalize pair: the transformation first, a why paragraph, then Covers and Mode bullets naming the serializer modules with their full titles * Label FunctionDefinition-normalized-whitespace an identity pin * Keep the trailing whitespace test bytes untouched; the drivers ignore both additions, so the spec count stays unchanged
Cover the tree-only grid emitter with a normalize pair: leading merge markers rebuild in trailing form and the pseudo rows re-emit in class, caption, body, footer order. * Add Table-normalized.tid with a class row, header cells, both merge forms, a rowspan, a trailing caption row and a footer row * Pin the emission verified by re-parse equivalence: |>|Cell9 | becomes |Cell9 |<| and the caption row moves below the class row
✅ Deploy Preview for tiddlywiki-previews ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Confirmed: pmario has already signed the Contributor License Agreement (see contributing.md) |
📊 Build Size Comparison:
|
| Branch | Size |
|---|---|
| Base (master) | 2492.1 KB |
| PR | 2497.3 KB |
Diff: ⬆️ Increase: +5.2 KB
⚠️ Change Note Status
This PR appears to contain code changes but doesn't include a change note.
Please add a change note by creating a .tid file in editions/tw5.com/tiddlers/releasenotes/<version>/
📚 Documentation: Release Notes and Changes
💡 Note: If this is a documentation-only change, you can ignore this message.
Member
Author
|
@linonetwo ... Have a closer look here. It should pretty much do round trips with every valid wikitext, that is available in data/ driven tests. I did create several new tests and could use all existing other tests too. |
Give quoteblock, styleblock, html and conditional their own it() blocks so a failure names the culprit rule; the five expectations are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changed files at a glance. This experimental PR contains some core changes, that will be separated. See #9908
plugins/tiddlywiki/wikitext-serialize/rules/plugins/tiddlywiki/wikitext-serialize/utils/parsetree.jscore/modules/parsers/wikiparser/wikiparser.jstrims block spans and stamps the block position. The ten rule files add small annotations, fix the terminator bugs and wrap hard linebreak regions. Each file belongs to one sub-issue below.editions/test/tiddlers/tests/*.jstest-wikitext-serialize.jsruns four groups of tests. Six new spec files cover the core changes, one file per change. Each of them fails without its fix.editions/test/tiddlers/tests/data/serialize/Inputagainst anExpectedWikitext. Every file has a one line description. The pairs also contain aninfotiddler that explains the test.A side effect of these experiments is a mechanism, that will allow us to create a "pretty printer" for TW wikitext.