Markdown syntax for roles and directives

[choldgraf]

Last week we had a few nice conversations around "how to extend Markdown to support roles and directives from Sphinx".

This is a quick issue to try an keep track of our thinking there.

What kinds syntax chunks are there?

• operate on raw text and yield a block element
• operate on raw text and yield an inline element
• operate on commonmark block content and yield a block element
• operate on commonmark inline content and yield an inline element

What we arrived at

After a few conversations, we arrived at a syntax that uses triple-backticks, followed by directive name, followed by configuration with two options (either using {key=val} or YAML front-matter inside the code block).

So something like (ignore the slashes, just for rendering purposes):

\```mydirective {key=val}
\```

And for in-line text, using single-backticks followed by an identifier in the traits associated w/ it:

This is `my role`{myrolename key=val}

This effectively treats everything as "raw text", with the idea that this would degrade gracefully by just rendering as a raw blob if the directive didn't exist.

How would this clash with current markdown or rST behavior?

Something like this:

• If a triple-backtick block is found, check the language associated with it
• See if the language exists in a list of directives in the current environment
  • If so, treat it as a directive block and not a language block. Anything in {} becomes configuration for the directive. Anything inside the backticks becomes content that is processed by the directive.
  • If not, treat it as a language block

Something similar could be done with in-line blocks

What others have found

• here is a good thread describing how pandoc approaches this
• here is how autostructify handles it
• here is a nice proposal for how to do this in commonmark

[choldgraf]

A suggestion from John Macfarlane:

Sincewe've been talking about dedicated syntax that would map on to a directive, but wouldn't be confusable with code blocks, use what RMarkdown and Pandoc do and use {} for "special" inline or block literals, something like:

```{mydirective}
This is
my special section
literal
```

We could assume that any code blocks that had curly brackets were block-level directives, and reference the first element in the {} against our list of directives. If it doesn't exist, fall back to assuming it is just an attribute.

This would also be fairly parsable in other markdown parsers, since the {} pattern is quite common, and we wouldn't introduce any extra syntax. Also we could then still use

```language
This is
my language syntax
```

[choldgraf]

also - just a note for @rowanc1 here, I feel like if we end up using Sphinx and have a directive / role syntax for markdown, then maybe that's a place where components.ink pieces could be inserted into content at build time by writing a role/directive that injects the proper JS and HTML into the page (maybe as a separate sphinx extension?) curious what you think about that...

[rowanc1]

My interpretation of how this would apply is:

Some happy text.
```{ink-scope name=scope1}
``{ink-var name=x value=2}
My variable $x=$``{ink-display name=x}.

```

I am putting the scope in there as an example. It gets a bit messy, especially if you have multiple block directives. For example, styling an input as a callout box.

A couple of questions:

• Would indentation or raw html input be allowed?
• Any thoughts on empty content for inline elements?

For example: indentation

{ink-scope name=scope1}:
    ``{ink-var name=x value=2}
    {ink-callout kind=info}:
        Variable $x=$``{ink-display name=x}.

For example: html

<ink-scope name="scope1">
    ``{ink-var name=x value=2}
    Variable $x=$``{ink-display name=x}.
</ink-scope>

And that would either be ignored in other representations - or perhaps if you have an intermediate AST then it could last until there? I liked the comment you posted about the C markdown parser coming to a common xml representation that can be acted upon.

[chrisjsewell]

Behold the first Markdown directive parser!

See the bottom of https://github.com/chrisjsewell/mistletoe/blob/myst/test.ipynb

Current format is:

````{note}
abcd *abc* [a](link)

```{warning}
xyz
```

````

which is transformed to docutils AST:

<document source="">
    <note>
        <paragraph>
            abcd 
            <emphasis>
                abc
             
            <pending_xref refdomain="True" refexplicit="True" reftarget="link" reftype="any" refwarn="True">
                <reference refuri="link">
                    a
        <warning>
            <paragraph>
                xyz

FYI for all the tests (which are extensive) see: https://travis-ci.org/chrisjsewell/mistletoe

[choldgraf]

[akhmerov]

This looks totally awesome!

A quick question: should it be possible to configure default directive/role akin to sphinx? These could use blank {} for example.

[jstac]

Very cool. I see you're picking up line numbers corresponding to cells in the AST. So ticking all the boxes already in terms of what was needed...

[chrisjsewell]

So I've added testing against most of the docutils directives (see here), and added parsing of arguments, e.g.

```{image} path/to/image
```

The last part is to parse options. It has been mentioned about parsing like ```{name key=value}, but a major problem with this is it would break the current code fence regex, which looks for a string with no spaces for the language component (I also don't think it looks very nice).

I think the YAML block is the best way and I was thinking, for efficient parsing, it would be good to signify in the first line if the block contains options. Something like:
(note the +)

```{image}+ path/to/image
height: 20
width: 40
---
Here is a *caption*.
```

Then it would read everything as YAML until either a --- is found or the end of the block is reached.

[jstac]

You've worked hard!!!

I personally find the YAML syntax far more readable than {name key=value} when there are multiple options. But opinion will be split on that point.

Regarding the YAML syntax, could you do

```{image} path/to/image
---
height: 20
width: 40
---
Here is a *caption*.
```

That seems a bit more symmetric --- and hence easy to remember.

[chrisjsewell]

Yeh that could also work ta.

Implemented roles and math as well now (no option key/val parsing yet). It actually could end up being more powerful than RST in some respects, because you can nest inline elements, which isn't possible in RST:

````{note}
abcd *abc* [a](link)

```{warning}
xyz
```

````

```{figure}+ path/to/image
height: 40
---
Caption
```

**{code}`` a=1{`} ``**

**$a=1$**

$$b=2$$

`` a=1{`} ``

goes to:

<document source="">
    <note>
        <paragraph>
            abcd 
            <emphasis>
                abc
             
            <pending_xref refdomain="True" refexplicit="True" reftarget="link" reftype="any" refwarn="True">
                <reference refuri="link">
                    a
        <warning>
            <paragraph>
                xyz
    <figure>
        <image height="40" uri="path/to/image">
        <caption>
            Caption
    <paragraph>
        <strong>
            <literal classes="code">
                a=1{`}
    <paragraph>
        <strong>
            <math>
                a=1
    <paragraph>
        <math_block xml:space="preserve">
            b=2
    <paragraph>
        <literal>
            a=1{`}

[chrisjsewell]

@choldgraf @mmcky @jstac @AakashGfude I've added the Sphinx Parser 😃

You just install my fork of mistletoe (pip install -e .[sphinx,testing], on the myst branch), and add extensions = ["mistletoe"] to your conf.py and it will pick up all the .md files.

Note if you look in myst/test/test_sphinx/test_sphinx_builds.py, I have set up automated testing of sphinx builds, for folders in myst/test/test_sphinx/sourcedirs. So if you run that with pytest it will actually generate the _build folders (comment out the remove_sphinx_builds fixture, so that they are not removed at the end of the test).

[chrisjsewell]

@choldgraf FYI front-matter does start with --- (see here), so it makes sense in the directives to also do this, which I've now changed to:

```{name} argument text
---
option: 1
---
content with *markdown* **syntax**
```

[jstac]

Love your work @chrisjsewell. Outstanding.

[choldgraf]

Duuude - it works! So cool! Tonight I'll try making a little sphinx documentation site in your myst branch using the content that @AakashGfude put together...I am curious how it'll look!

[jorisvandenbossche]

(Chris pointed me to those discussions; I am an extensive sphinx user due to being one of the maintainers of the pandas docs, which is a quite big sphinx site. And I am excited about the issues you are tackling here: I love sphinx, but I also love to see improvements to it ;))

One thing I am wondering: to what extent are you already set on the syntax for roles and directives?

It seems you are now taking the syntax for code (both for inline and blocks) with adding a role/directive name in the {}.

This is closer to existing markdown syntax, so I can imagine this is easier to extend an existing parser for this? (and it's also closer to things in the existing standard / pandoc, which are very good reasons)

But thinking about some usecases for roles in the documentation projects I am working with, and I think something along the lines of the generic directives syntax proposal might be easier to work with (as an end user):

Small example rst snippet:

We can link to :meth:`pandas.DataFrame` in the API reference
or to another section :ref:`here <label>` (:issue:`1234`).

How it might look like based on the role examples above (the details might not be correct):

We can link to `pandas.DataFrame`{meth} in the API reference
or to another section `here`{ref, id=label} (`1234`{issue}).

And how it might look like with the linked proposal:

We can link to :meth[pandas.DataFrame] in the API reference
or to another section :ref[here]{label} (:issue[1234]).

Personally, I think the third snippet "looks" better than the second (but that's very subjective of course. Maybe that's because I am so used to having colons in rst .. ;-))
But maybe a slightly more objective argument: I think having the role name come first, instead of in the end, improves readability. And it also gives more contrast with actual code snippets.