How best to work with Mistletoe?

[chrisjsewell]

In our review of existing Markdown parsers (ExecutableBookProject/meta#18 ExecutableBookProject/meta#19) mistletoe came out as the favourite for its mix of parsing speed, extensibility and well constructed API. However, the one con was that it is not really being actively maintained.

To this extent, for current development, a fork is being used as a dependency (ExecutableBookProject/mistletoe#1), to allow for required/desired changes to the core code. Eventually though these need to be up-streamed, or the fork needs to become its own package (with pypi and conda distributions).

What is the best way to contact/work with the current maintainer to achieve this?

[choldgraf]

I think the first step would be to try making a few small-ish PRs and see how responsive they are. We can slowly try to close the gap between this fork and mistletoe master, but if we make a giant PR all at once it'll probably be much less-likely they'll engage.

In an ideal world, we'd get the keys to mistletoe but it may be some time before that happens depending on what kinda maintainer they are. In an extreme case, if we decide it's not worth engaging w/ a fork, then we can hard-fork it and rename the project to something else.

WDYT?

[chrisjsewell]

I imagine making a PR will just hit a brick wall, like the other pending PRs. But can give it a go.

Yeh it depends on the reason why he is currently not engaging. If its just that he's super busy with other things, it might be good to send him a nicely worded email at some point, and explain our position and that we would (maybe) be happy to pick up some of the slack on maintenance.

[choldgraf]

yep for sure - I think that either way though we've gotta start by making a (probably small) PR...if we decide to escalate one way or another, then we can say that we've gone through the typical ways of engaging first

[chrisjsewell]

Issue opened miyuchina/mistletoe#92

[choldgraf]

Shall we open up a PR in mistletoe? We haven't heard anything back from the author since our original post...perhaps a minor improvement, or the bugfix mentioned in #102 ? Just wanna keep the heartbeat going there...

[choldgraf]

Another note - Mistletoe was announced on Reddit 2 years ago here:

https://www.reddit.com/r/Python/comments/6sjz3x/mistletoe_a_fast_extensible_markdown_parser_in/

[chrisjsewell]

Shall we open up a PR in mistletoe?

See miyuchina/mistletoe#94

[choldgraf]

nice - thanks

[chrisjsewell]

FYI, here is a working list of 'up-streamable' changes that would be desired:

• Move testing from unittest to pytest: pytest is now the de facto testing architecture and vastly improves the usability/flexibility of testing. (already introduced in fork)
• Introduce pre-commit code linting and formatting: This standardizes the code style across the package, and ensures that new commits and Pull Requests also conform to it. (already introduced in fork)
• Introduce ReadTheDocs documentation
• Add a conda-forge distribution of the package
• Improve the AST API and documentation: I view panflute's implementation of the pandoc API in python, as the gold standard for how a pythonic AST API should be written and documented. Some tweaks to the current token class objects, and creating auto-generated RTD documentation, could achieve this.
• Storage of source line/column ranges: LSP and good rendering reporting of warnings/errors, requires access to the source line and column ranges of each parsed token.
• Asynchronous parsing: LSP requires documents to be parsed asynchronously. Currently, mistletoe contains a number of global state objects, which make parsing inherently not thread-safe. The simple solution to this is to store these items as threading.local objects. A related but slightly more complete solution is to introduce the idea of a 'scoped session', similar to that used by sqlalchemy for database access: Contextual/Thread-local Sessions
• Improve extensibility of block tokens: A Markdown parser is inherently a Finite State-Machine + Memory (a.k.a Push-down Automata (PDA)), with parsing tokens as states (for a good example of a python state-machine see pytransitions). The problem with extensibility, is that inherently states are interdependent; when introducing a new state/token you must provide logic to all the other tokens, w.r.t to when to transition to this new token. Currently, MyST Parser sub-classes nearly all the Mistletoe block tokens to implement the extensions it requires, but it would be ideal if there was a more systematic approach for this.
• Improve extensibility for span tokens: Mistletoe does allow for span token extensions to be added, at least in a simple way. However, as with block tokens above, there is often an interconnectivity to them, especially when considering nested span tokens. As of 7cc2c92, MyST-Parser now overrides some of Mistetoe's core logic to achieve correct parsing of Math tokens, but if possible this should be made more general.
• Improve rendering logic: Currently, there is no concept of recursive walk-throughs or 'visitor' patterns in the Misteltoe BaseRenderer, which is a better method for rendering tree like structures (as used by docutils/panflute). Also, the current token instantiating (within context managers) needs improvement (see Document side-effects of renderers' initialisation miyuchina/mistletoe#56).

[choldgraf]

Nice - this is really helpful.

I think the other question we should all consider is: what happens if it just doesn't work out? I am curious what @chrisjsewell thinks about the maintainability and sustainability of caring for Mistletoe (or in a worst-case scenario, needing to care for a differently-named fork of it). After having worked with the codebase thus far, what are your thoughts on that?

[chrisjsewell]

As you've noticed, I got impatient lol. mistletoe-ebp now has CD on pypi and soon on conda-forge. Then I can propagate this to MyST-Parser.

I still think mistletoe is the right tool for the job, in terms of its base API, it just needs the improvements above. I don't think maintainability/sustainability should be too much of an issue, given that a robust test suite is in place, since the underpinning Markdown syntax isn't exactly going to be changing any time soon (if ever from what we've seen from the decade long forum conversations lol).