Aho-Corasick Freelinks Enhancement for Large Wikis and Non-Latin Titles#9084
Conversation
Replaces regex with Aho-Corasick, adds chunking (100 titles/chunk), cache toggle, and Chinese full-width symbol support. Tested with 11,714 tiddlers.
|
@s793016 It appears that this is your first contribution to the project, welcome. With apologies for the bureaucracy, please could you prepare a separate PR to the 'tiddlywiki-com' branch with your signature for the Contributor License Agreement (see contributing.md). |
✅ Deploy Preview for tiddlywiki-previews ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
| An override of the core text widget that automatically linkifies the text, with support for non-Latin languages like Chinese, prioritizing longer titles, skipping processed matches, excluding the current tiddler title from linking, and handling large title sets with Aho-Corasick algorithm and fixed chunking (100 titles per chunk). Includes optional persistent caching of Aho-Corasick automaton, controlled by $:/config/Freelinks/PersistAhoCorasickCache. | ||
|
|
||
| \*/ | ||
| (function(){ |
There was a problem hiding this comment.
On master branch we already remove, you need to let AI take latest code as reference. You can drag several js file to AI context. And don't remember to let it follow ES5 and other tw code style.
(function(){
/*jslint node: true, browser: true */
/*global $tw: false */
There was a problem hiding this comment.
Sorry I'm a noob, I only use forums when I use github.
/*jslint node: true, browser: true */
/*global $tw: false */
deleted
There was a problem hiding this comment.
Modified it again and it's now functioning properly
| } | ||
|
|
||
| // Aho-Corasick implementation with enhanced error handling | ||
| function AhoCorasick() { |
There was a problem hiding this comment.
refactor this class to another js, and require it in this file instead.
There was a problem hiding this comment.
move AhoCorasick to AhoCorasick.js ok
There was a problem hiding this comment.
Modified it again and it's now functioning properly
| // Get the current tiddler title | ||
| var currentTiddlerTitle = this.getVariable("currentTiddler") || ""; | ||
| // Check if persistent caching is enabled | ||
| var persistCache = self.wiki.getTiddlerText(PERSIST_CACHE_TIDDLER, "no").trim() === "yes"; |
There was a problem hiding this comment.
no need to trim, like other config tiddlers.
There was a problem hiding this comment.
Modified it again and it's now functioning properly
| this.wiki.getPersistentCache(cacheKey, function() { | ||
| return computeTiddlerTitleInfo(self, ignoreCase); | ||
| }) : | ||
| this.wiki.getGlobalCache(cacheKey, function() { |
There was a problem hiding this comment.
why use both getPersistentCache and getGlobalCache?
There was a problem hiding this comment.
Cache logic clarification
Keeps the two caching policies, but adds a note saying “Unified Caching Policy”
This is actually a meaningful design:
getPersistentCache: cross-session persistence, suitable for large wikis
getGlobalCache: single-session cache, suitable for general use.
Selected dynamically based on $:/config/Freelinks/PersistAhoCorasickCache configuration.
There was a problem hiding this comment.
Oh, I search the repo, there is no getPersistentCache, so this is Grok's illusion. So there is only getGlobalCache. While I hope we cound have a getPersistentCache in indexed-db, but that is for another PR.
There was a problem hiding this comment.
No, “getPersistentCache“ is not an illusion, it's an additional static caching of large data (default: off).
Persistent Cache Toggle:
Optional persistent caching of the Aho-Corasick automaton via $:/config/Freelinks/PersistAhoCorasickCache (default: no).
Enabled (yes) reduces rebuild time by ~100-200ms per refresh.
Cache invalidates on title changes, ensuring consistency.
There was a problem hiding this comment.
The global cache is very short lived. Everytime the wiki.addTiddler() function is called the global cache will be cleared. So basically whenever there is any UI interaction. Since UI interactions most to the time use state- or temp-tiddlers
Line 1232 in a3b8c7c
Also see: freelinks.js uses globalCache object, which is destroyed most of the time #4572
| titleChunks.push(titles.slice(i, i + chunkSize)); | ||
| } | ||
| // Log chunking details for debugging | ||
| console.log("Total titles:", titles.length, ", Chunk size:", chunkSize, ", Number of chunks:", titleChunks.length, ", Persistent cache:", self.wiki.getTiddlerText(PERSIST_CACHE_TIDDLER, "no").trim()); |
There was a problem hiding this comment.
remove logs before merging.
There was a problem hiding this comment.
Modified it again and it's now functioning properly
| // Add remaining text | ||
| if(currentPos < text.length) { | ||
| newParseTree.push({ | ||
| type: "plain-text", |
There was a problem hiding this comment.
do we have this kind of parse tree node? I haven't encounter it before.
There was a problem hiding this comment.
@linonetwo Thanks for the feedback! I tried changing plain-text to text, but it caused TiddlyWiki to fail starting, likely due to parser conflicts. I've reverted to plain-text for now to ensure stability.
Signed Contributor License Agreement for my first contribution (PR TiddlyWiki#9084).
|
@s793016 Please add this plugin to tw5-com edition, so we could try it in the preview site. See https://github.com/TiddlyWiki/TiddlyWiki5/pull/8991/files#diff-c0054677a142c9f59b544336b05ce6bcf927594230e045f290e2b2eaf2a8409c for example of adding core plugins. |
| this.parentDomNode = parent; | ||
| this.computeAttributes(); | ||
| this.execute(); | ||
| this.renderChildren(parent, nextSibling); |
There was a problem hiding this comment.
TW uses tabs for indentation. These spaces need to be replaced.
| text: this.getAttribute("text", this.parseTreeNode.text || "") | ||
| }]; | ||
| // Only process links if not disabled and we're not within a button or link widget | ||
| if (this.getVariable("tv-wikilinks", { defaultValue: "yes" }).trim() !== "no" && |
There was a problem hiding this comment.
indentation needs to be checked here too
| An override of the core text widget that automatically linkifies the text, with support for non-Latin languages like Chinese, prioritizing longer titles, skipping processed matches, excluding the current tiddler title from linking, and handling large title sets with Aho-Corasick algorithm and fixed chunking (100 titles per chunk). Includes optional persistent caching of Aho-Corasick automaton, controlled by $:/config/Freelinks/PersistAhoCorasickCache. | ||
|
|
||
| \*/ | ||
| (function() { |
There was a problem hiding this comment.
This function() wrapper is not needed anymore. We removed it from the whole repo
|
|
||
| exports.text = TextNodeWidget; | ||
|
|
||
| })(); |
There was a problem hiding this comment.
same here -- This belongs to the function() wrapper. Should not be needed any more
📊 Build Size Comparison:
|
| Branch | Size |
|---|---|
| Base (master) | 2527.7 KB |
| PR | 2532.2 KB |
Diff: ⬆️ Increase: +4.4 KB
| for TiddlyWiki freelinking functionality. | ||
|
|
||
| \*/ | ||
| (function(){ |
There was a problem hiding this comment.
We don't need this. You need to provide latest code for AI. Seems it doesn't know we already don't use this?
Also seems they don't use jsdoc for now, those /**. I like jsdoc, but currently it don't exist in the repo.
Integrate text.js and aho-corasick.js into tw5-com per @linonetwo's request in PR TiddlyWiki#9084.
Integrate text.js and aho-corasick.js into tw5-com per @linonetwo's request in PR TiddlyWiki#9084.
add freelink into editions/tw5.com/tiddlywiki.info ok |
| "tiddlywiki/internals", | ||
| "tiddlywiki/menubar", | ||
| "tiddlywiki/railroad", | ||
| "tiddlywiki/tour" |
There was a problem hiding this comment.
missing a comma here. Maybe you should use VSCode to edit, this kind of error will have visual warning on edit time. But anyway, just a small error.
There was a problem hiding this comment.
It might be editions/prerelease/tiddlywiki.info , sorry @s793016 , I'm also confused about this.
There was a problem hiding this comment.
try add it to editions/prerelease for testing ok
|
@saqimtiaz |
plugins/tiddlywiki/freelinks/text.js#L25 [ESLint PR code] reported by reviewdog 🐶 Strings must use doublequote.
plugins/tiddlywiki/freelinks/aho-corasick.js#L193
[ESLint PR code] reported by reviewdog 🐶
Strings must use doublequote.
Raw Output:
{"ruleId":"@stylistic/quotes","severity":2,"message":"Strings must use doublequote.","line":193,"column":50,"nodeType":"Literal","messageId":"wrongQuotes","endLine":193,"endColumn":52,"fix":{"range":[5743,5745],"text":"\"\""}}
|
Thanks @s793016 @linonetwo @saqimtiaz |
|
[deleted post] |
|
@twMat ... can you please create a new issue, so we can properly track it. comments on merged PRs are usually gone from the overviews and are very likely to be forgotten |
|
@twMat ... did you start from https://tiddlywiki.com/prerelease - download empty and installed the plugin from there. I do not get the wrong rendering in the plugin tiddler content tab |
|
[in reference to my deleted post, two posts up] @pmario - I'm embarrassed to say that I hadn't updted wiki that I tested the plugin in so, unsurprisingly, it didn't work. Thank you for replying and sorry for wasting peoples time. EDIT: mierde... there does seem to be some bug after all. I'll post an issue. |
…es (TiddlyWiki#9084) * Enhance Freelinks with Aho-Corasick for long titles and large wikis Replaces regex with Aho-Corasick, adds chunking (100 titles/chunk), cache toggle, and Chinese full-width symbol support. Tested with 11,714 tiddlers. * delete comment * Create AhoCorasick.js * Update text.js * Update text.js * Update AhoCorasick.js * Update text.js * Update text.js * move AhoCorasick to AhoCorasick.js * update AhoCorasick.js * Delete core/modules/utils/AhoCorasick.js wrong place * Update text.js * indentation modify * remove function {} * remove function {} * Rename AhoCorasick.js to aho-corasick.js correct filename * Update tiddlywiki.info add freelink * missing a comma here * clean up comments & use old style * try add it to editions/tw5.com-server for testing * try add it to editions/prerelease for testing * optimized * optimized * add setting for "Persist AhoCorasick cache" * add dynamic limits * remove comment * revert to 5f0b98d * try sort alphabet * try sort alphabet * try sort alphabet * typo freelink -> freelinks * typo freelink -> freelinks * typo freelink -> freelinks * Update readme.tid * Update aho-corasick.js Dynamically adjust limit parameters to avoid problems caused by hard-coded limits. * Update text.js Dynamically adjust limit parameters to avoid problems caused by hard-coded limits. * Update tiddlywiki.info remove other plugin for test plugin conflict * Update tiddlywiki.info * Update tiddlywiki.info * Update aho-corasick.js Description of major changes Improve state transition logic - Ensure to go back to root node correctly in case of mismatch, and check root node for current character transition Fix failed link traversal - Add condition node ! == this.trie to avoid infinite loop at root node Enhance output collection - collect output not only from current node, but also from all nodes on failed link path, which is key to Aho-Corasick algorithm Add safety limit - collectCount < 10 to prevent failed link loops Translated with DeepL.com (free version) * Update aho-corasick.js Word Boundary Check - The isWordBoundaryMatch function checks if the match is on a word boundary: Alphanumeric characters [a-zA-Z0-9_] are regarded as unicode characters At least one non-unicode character must be present before and after the match for it to be considered valid. * Update text.js Word Boundary Check - The isWordBoundaryMatch function checks if the match is on a word boundary: Alphanumeric characters [a-zA-Z0-9_] are regarded as unicode characters At least one non-unicode character must be present before and after the match for it to be considered valid. * Update settings.tid Word Boundary Check - The isWordBoundaryMatch function checks if the match is on a word boundary: Alphanumeric characters [a-zA-Z0-9_] are regarded as unicode characters At least one non-unicode character must be present before and after the match for it to be considered valid. * fix Word Boundary logic * remove PersistentCache @ text.js * remove PersistentCache @settings.tid * Update readme.tid for Word Boundary Check * Update aho-corasick.js Organize and delete comments * Initial commit of freelinks plugin * Update settings.tid Organize and delete comments * Update tiddlywiki.info add back other plugin * Update tiddlywiki.info alphabet sort * Update readme.tid for new future The plugin supports non-Western language tiddler titles (e.g., Chinese) and prioritizes longer tiddler titles for matching, ensuring accurate linking in diverse contexts. Furthermore, the current tiddler title within its own content is excluded from generating links to avoid self-referencing. * Update readme.tid * Update plugins/tiddlywiki/freelinks/text.js Co-authored-by: Mario Pietsch <pmariojo@gmail.com> * Update plugins/tiddlywiki/freelinks/aho-corasick.js Co-authored-by: Mario Pietsch <pmariojo@gmail.com> * Update plugins/tiddlywiki/freelinks/aho-corasick.js Co-authored-by: Mario Pietsch <pmariojo@gmail.com> * Update plugins/tiddlywiki/freelinks/text.js Co-authored-by: Mario Pietsch <pmariojo@gmail.com> * Update plugins/tiddlywiki/freelinks/aho-corasick.js Co-authored-by: Mario Pietsch <pmariojo@gmail.com> * Update text.js Added locale configuration support - Added LOCALE_CONFIG_TIDDLER constant to make the sorting locale configurable instead of hardcoded "zh" Optimized title processing - Combined the filtering and escaping logic into a single pass to reduce duplication Added trim() for ignoreCase - Applied .trim() to the ignore case variable for consistency Enhanced refresh logic - Added locale configuration tiddler to the refresh check Improved comments - Added explanation for why sorting is necessary (prioritizing longer titles) * Update text.js we don't need to specify 'zh' at all * Update aho-corasick.js This single line change would add support for: Accented letters: á, é, í, ó, ú, à, è, ì, ò, ù, ä, ë, ï, ö, ü, ñ, ç, etc. Most Western European languages (Spanish, French, German, Italian, Portuguese, etc.) * Update aho-corasick.js useage * Update readme.tid for Writing Style * Update tiddlywiki.info revert all the changes * Update tiddlywiki.info revert all the changes * Update tiddlywiki.info revert all the changes * Update tiddlywiki.info revert * Update text.js plugins/tiddlywiki/freelinks/text.js#L25 [ESLint PR code] reported by reviewdog 🐶 Strings must use doublequote. * Update aho-corasick.js plugins/tiddlywiki/freelinks/aho-corasick.js#L193 [ESLint PR code] reported by reviewdog 🐶 Strings must use doublequote. Raw Output: {"ruleId":"@stylistic/quotes","severity":2,"message":"Strings must use doublequote.","line":193,"column":50,"nodeType":"Literal","messageId":"wrongQuotes","endLine":193,"endColumn":52,"fix":{"range":[5743,5745],"text":"\"\""}} --------- Co-authored-by: Mario Pietsch <pmariojo@gmail.com>
Make up for my missing change note TiddlyWiki#9084
remove those AI signatures
Make up for my missing change note TiddlyWiki#9084
Make up for my missing change note TiddlyWiki#9084
| this.patternCount = 0; | ||
| }; | ||
|
|
||
| AhoCorasick.prototype.getStats = function() { |
There was a problem hiding this comment.
@s793016 @linonetwo are we sure that this method has the correct logic?
In countNodes() we increment the local variable patternCount, but we never use it and return this.patternCount instead.
There was a problem hiding this comment.
Simply put: it counts one variable but returns another, effectively rendering the counting useless.
This eslint-disable-next-line no-unused-vars annotation itself hints at the problem—ESLint has already warned that the local variable patternCount is unused.
The getStats() function's return value is actually correct—it's just that the intermediate calculation is redundant.
Additionally, thanks to Claude Sonnet 4.6, we've identified other issues that will be addressed collectively.
There was a problem hiding this comment.
@s793016 Now you see AI are lazy, they tend to hide the error using eslint-disable, or modify the test case so its work could easily done. Make sure to use Claude Opus 4.6 to final review the code.
There was a problem hiding this comment.
I think it's not that AI is lazy, but that the AI at the time wasn't strong enough. The first version before June 2025 was clumsily put together by Grok, then to extract the algorithm independently, Grok couldn't handle it and passed the baton to Claude, and later I showed the code to ChatGPT, who also didn't spot the issues. Until a couple of days ago when Claude 4.6 came online, it immediately identified several logic problems—the progress of AI in this past half year or more has really been tremendous.
Back in mid-2025, the models (including earlier Groks) were still hitting limits on complex refactors like extracting algorithms without introducing subtle logic bugs. Handing it off to Claude made sense at the time, and even ChatGPT missing those issues highlights how context-dependent error detection can be. Fast-forward to now (early 2026), and yeah, the jumps in reasoning depth are huge. Claude 4.6 spotting those logic flaws right away? That's a testament to better multi-step thinking and pattern recognition in newer architectures. We've seen similar strides across the board.
Make up for my missing change note TiddlyWiki#9084
release: 5.4.0
This PR enhances the Freelinks plugin (
plugins/tiddlywiki/freelinks/text.js) with Aho-Corasick for faster title linking, optimized for large wikis (11,714+ tiddlers) and non-Latin languages like Chinese. Developed for my 13,000-tiddler wiki, it addresses long titles and full-width symbols (e.g.,:, U+FF1A).Changes
$:/config/Freelinks/PersistAhoCorasickCache).wikilink.js.Performance
mainRefresh2.4-4.6s, linkification 100-500ms.Testing
$:/config/Freelinkstoyes,$:/config/wikilinkstono.[[MyCamelCaseTiddler]] 和 [[雪狼湖:活動]].Motivation
My wiki relies on Freelinks for long Chinese titles (e.g., “雪狼湖:活動”) in read-only content like student submissions. This addresses performance issues with long titles, as noted by @linonetwo, making Freelinks viable for similar wikis.
Related discussion: [link to discussion, e.g., https://github.com/Jermolene/TiddlyWiki5/discussions/XXXX]
@linonetwo @Jermolene