Migrate modules docs to markdown

[roberth]

Project description

Migrate module docs to markdown.

Pandoc should be able to mix and match 🤞

Suggested pipeline

• XML
  • -> custom XSLT ->
• Markdown with arbitrarily formatted snippets
  • -> pandoc ->
• anything.

Suggested migration path

1. Implement the pipeline, so we support multiple languages
2. Start the conversion to description = lib.docMD "...", etc
3. Forbid non-wrapped description (and other such attrs)
4. Switch the default to markdown, remove the lib.docMD calls.
5. Optionally, simplify the pipeline

From the broader ecosystem perspective, it would be good to keep the flexible pipeline, as we can use it to dismantle the cottage industry of custom module docs generators. It's a matter of wrapping it with the right pandoc syntax.

Technical details

(just to get a feel for it)

description = lib.docDB "This is <literal>docbook</literal>.";

->

description = { type = "doc"; lang = "docbook"; text = "This is <literal>docbook</literal>."; };

->

Some XML

->

Markdown:

    Description:

    ```{=docbook}
    This is <literal>docbook</literal>.
    ```

Alternatives

• Maybe XML + XSLT should be JSON + Python. @pennae has already converted some things to python as part of their eval performance work.

Metadata

• homepage URL: module system does not have a dedicated homepage
• source URL: lib/, nixos/, pkgs/
• license: mit, bsd, gpl2+ , ...
• platforms: unix, linux, darwin, ...

[jtojnar]

It would be nice but there was opposition to including pandoc in the NixOS system closure. So we would either need to decouple the manual from the system, choose some lighter parser (maybe lowdown + HTML→DocBook via XSL), or cache pre-generated DocBook like we currently do with the NixOS manual.

[roberth]

opposition [to increase in build closure size because of pandoc]

Sure! If we don't want pandoc in the build closure for NixOS, we can choose to implement step 5. NixOS' dependency on pandoc will be a temporary affair.

lowdown + HTML→DocBook via XSL

We'll need a tool that can convert tons of snippets at pace. Pandoc can do this (as linked before). Shelling out will not be an option, so I suspect that this would involve a custom tool. Could be worth it.

cache pre-generated DocBook

This is terrible developer UX. If we want people to write less docs, this is one way to do it.
It will be even more complicated, because or custom tooling can't process this file-by-file like we can in the manual.

[pennae]

Maybe XML + XSLT should be JSON + Python. @pennae has already converted some things to python as part of their eval performance work.

probably, yeah. there's not that much literal docbook in nixos modules that's properly labelled as such, and among the great many docbook-using descriptions that are not properly labelled by far the most use for docbook uses only a few tags (literal, option, filename, programlisting, ...). parsing descriptions as xml fragments and running a custom transform over them actually looks quite feasible!

[roberth]

Hmm yeah, that makes me think that perhaps my migration path is overly correct. We could just do it and fix the occurrences of the small set of tags used after switching.

parsing descriptions as xml fragments and running a custom transform over them actually looks quite feasible!

I would avoid writing any and all parsing logic, and only go from semi-structured data to {markdown + pandoc raw attribute} in our own tooling.
It's not worth writing and maintaining a complex solution just to have ok-looking docs during the migration. Of course this is subjective, but the temporary nature is evident. If such a custom transform is a migration tool rather than an integral part of the docs pipeline, I'm all for it. That also takes the pressure off its correctness, as we can fix a few things up by hand.

[zimbatm]

I bet it's possible to introduce and get rid of pandoc until the next release. It would be a shame to block the refactor for a temporary dependency, as long as it doesn't stay around forever.

[jtojnar]

We do not have a time frame, so it might take longer than the next release. Especially when we have not even settled on a toolchain yet (currently the candidates are #105036 and #108063). And people who would be annoyed by the closure do often use unstable.

So, while deciding that having them choose between having pandoc in their system closure and disabling local docs temporarily can be a reasonable trade off for the long term goal of improving the docs infrastructure, it might still turn out few sed invocations is actually a better interim solution.

[edolstra]

I would keep it simple and just automatically convert all description strings to Markdown in one commit, so without introducing docMD, and drop support for Docbook descriptions immediately. That way we don't need pandoc in the system closure.

P.S. NixOS/nix#6583 already assumes that option descriptions are in markdown. Supporting other markup languages would just make life more complicated.

[pennae]

we've made an attempt at producing a script to migrate docs.

perl -pi -e 's,<literal>([^`]*?)</literal>,`$1`,g' nixos/modules/**/*.nix
perl -pi -e 's,<replaceable>([^»]*?)</replaceable>,«$1»,g' nixos/modules/**/*.nix
perl -pi -e 's,<filename>([^`]*?)</filename>,`$1`,g' nixos/modules/**/*.nix
perl -pi -e 's,<code>([^`]*?)</code>,`$1`,g' nixos/modules/**/*.nix
perl -pi -e 's,<option>([^`]*?)</option>,`$1`,g' nixos/modules/**/*.nix
perl -pi -e 's,<command>([^`]*?)</command>,`$1`,g' nixos/modules/**/*.nix
perl -pi -e 's,<link xlink:href="(.+?)" ?/>,$1,g' nixos/modules/**/*.nix
perl -pi -e 's,<link xlink:href="(.+?)">(.*?)</link>,[$2]($1),g' nixos/modules/**/*.nix
perl -pi -e 's,<package>([^`]*?)</package>,`$1`,g' nixos/modules/**/*.nix
perl -pi -e 's,<emphasis>([^*]*?)</emphasis>,*$1*,g' nixos/modules/**/*.nix

perl -pi -e 'BEGIN{undef $/;} s,<citerefentry>\s*<refentrytitle>\s*(.*?)\s*</refentrytitle>\s*<manvolnum>\s*(.*?)\s*</manvolnum>\s*</citerefentry>,`$1 ($2)`,smg' nixos/modules/**/*.nix

does a reasonable job of replacing most docbook tag with markdown constructs. around 2000 matches that look like xml tags remain after that, but a lot of that is either actual xml than can be safely ignored (eg fontconfig) or extended lists that have to converted with more thought anyway.

[pennae]

having spent a couple hours trying to hack together something to process the docbook xml snippets, it doesn't quite seem to be worth the trouble over using regexes. descriptions are not well-formed xml to begin with, interpolations split them up even further (and are a mess to track), some descriptions aren't even valid xml fragments—and the main benefit of automatic translation (being able to translate eg varlists) requires all of that to not be like it is. and we'll need sapient inspection of the results anyway. :/

[roberth]

descriptions are not well-formed xml to begin with, interpolations split them up even further (and are a mess to track), some descriptions aren't even valid xml fragments

This is why I proposed doing it as part of the existing pipeline, where we do have quality data.

Another option is to write a script that takes writes out a long markdown document with markers to delineate the individual descriptions, etc, split that up, and merge it back into the XML. This way, we don't need pandoc in the build closure.

I guess though, docs quality during the transition is not a priority, so then I suppose leaving some descriptions in a broken state is more acceptable than having pandoc in the closure?

---

Let me also mention literalDocBook, which occurs in example, defaultText. We'll want to have the same for markdown. At least these are easy to spot for the migration.