Improve documentation

[pitag-ha]

Current situation

As usual, we have different kinds of docs: a manual (outdated), the API documentation with integrated manual, a section about metaprogramming on ocaml.org, a couple of very simple examples, the dune docs on preprocessing specifications. Apart from that, very important resources to learn about how to write a ppx rewriter are a blog post and the compiler's parsing/parsetree.mli file.

The only resource that's kind of complete to learn how to write an extension node rewriter or a deriver is the blog post. The "manual" is more of a very high-level introduction to PPX (which is also very valuable but serves a different purpose) than a detailed manual. The API docs miss docs for some important values and aren't rendered perfectly. Apart from the docs for values/types, the API docs also include some important information about some of the modules (e.g. Ast_builder, Ast_pattern, Ast_traverse), which isn't easy to find and difficult to put into context.

How to improve the situation

For the resources that make sense to co-exist separately, we should make them all easy to find:

• Port the manual to odoc so that it appears on ppxlib's ocaml.org documentation next to its API docs.
• Link to the simple examples, the section on metaprogramming on ocaml.org, the dune documentation on preprocessing specifications, and the documentation on parsetree.mli the from within the beginning of the manual/documentation.
• Link to parsing.mli and to ast-explorer from within the manual section about the OCaml parsetree (see below).
• Link to the info about certain modules in the API docs from within the manual while putting that info into wider context.

Apart from those resources, we're currently also strongly relying on the blog post. Instead, we should integrate everything relevant in there (that is, most of it) into our manual and also write up some new bits for our manual:

• Make section about the OCaml AST more detailed. Re-use blog post and point to the compilers' parsing.mli and to ast-explorer.
• New section about how to construct an OCaml AST node, i.e. a about Ast_builder and metaquot. For metaquot we already have something in the manual.
• New section about how to de-construct the AST, i.e. Ast_pattern and metaquot. About Ast_pattern: To explain the general concept of Ast_pattern, we need to write something new; to explain the type parameters in Ast_pattern.t, we can re-use a blog post section; to explain its combinators, we can re-use a discuss answer and link to the module docs.
• Add subsection about attributes to the derivers section. Re-use blog post section.
• New section about locations. Re-use blog post. done here: https://ocaml.org/p/ppxlib/latest/doc/good-practices.html#resp_loc
• New section about how to report errors. @panglesd has already opened a PR for that.
• New short section about the other transformations one can write with ppxlib, i.e. general whole file transformations, linters, pre-transformations, instrumentations, etc, and point out in there the responsibility that comes with writing those kinds of transformations. We need to write that new.
• New section on how to write tests for PPXs. Maybe we can simply recommend dune cram tests?

And in our API docs, there are also some things that could be improved:

• Add documentation to all values and types that aren't generated (I'm excluding the generated ones since it would be time-intense and it's easier to explain their general pattern instead of documenting them individually)
• Add an introduction to the docs front page, in particular explaining how ppxlib is structured into the various libraries.
• Improve how the docs get rendered. That includes
  • We used to have a lot of Ppxlib__.Import.<some_type> types around in our docs (which lead to the situation that folks would use our internal Ppxlib__.Import API), which now I'm not seeing anymore. We should find out if that was fixed by odoc or if I'm just missing it now.
  • There are a couple of includes of modules that should either be hidden in the docs or, at least, by default be folded. E.g. the Ocaml_shadow include in the library interface module
  • The Selected_ast.Convert.To_ocaml (and, similarly, for Of_ocaml) module allows AST nodes to be converted between the AST version used by ppxlib and the AST version used by the compiler the project is compiled with. The latter is variable and depends on the project. So, if that isn't much work, it would be nice to represent that version in our docs by a place holder such as <compiler_version>:
    val copy_structure : Astlib.Ast_412.Parsetree.structure-> Astlib.Ast_<compiler_version>.Parsetree.structure
    Currently that version gets fixed by the compiler version odoc uses when generating the docs (apparently, 4.05):
    val copy_structure : Astlib.Ast_412.Parsetree.structure-> Astlib.Ast_405.Parsetree.structure

How to priorities

That's a long list of tasks, but lots of them only consist in moving info and adding links. Apart from that, I'd prioritize the new manual sections and adding missing API docs for the most important values. We can then complement the API docs for the more advanced/rare features over time.

[ComanderP]

I know this is an old issue and most things in the list have been done, but as someone who decided to write their first ppx a while ago, I have to say that the manual is nice but there could be a section dedicated to the parsetree (or maybe improving the ast builder/pattern documentation) as it's very hard to find any info about the ast for beginners. I wouldn't mind doing that myself but then again, I don't really know everything about the ast and there's not a lot of information online.

[ttzytt]

Hi, I'm a new learner of ppxlib, and here are some possible improvements I found about documentation. This issue was originally discussed in a post in discuss.ocaml.org.

Here, I'll use the documentation of Ast_helper.Exp as an example, but I found this to be quite common in the API part of the documentation.

I'd say that the doc does a good job listing all the APIs, including the type of arguments and return values. However, more explanation and examples would be really helpful.

For example, a short explanation can be added at the top of the page stating that the purpose of Ast_helper.Exp is to construct expressions in the AST, and some code examples with comments like this can be added (the code came from a modification of a tutorial written by @patricoferris in this post):

| { ptyp_desc = Ptyp_constr (lid, args); _ } ->
    (* We generate the name with "_stringify" on the end *)
    let new_name = Expansion_helpers.mangle_lid (Suffix "stringify") lid.txt in
    let new_lid = { lid with txt = new_name } in
    (* We turn it into an identifier hoping we have already generated 
       the function either by hand or by using the ppx *)
    let fn = Exp.ident new_lid in
    (* We construct an application of this function to an argument.
       Note that we assume the function may be higher-order if the type
       contains parameters e.g. int list *)
    let app =
      if args = [] then fn else
      Exp.apply fn (List.map (fun x -> Nolabel, expr_of_type x) args) 
    in
    [%expr fun x -> [%e app] x]

Also, I very much agreed with @ComanderP's opinion on adding a specific page to explain the overall design idea or a bigger picture of the AST. For example, I was able to grasp most of the meaning of the AST by looking at this page, but one thing that resulted in some confusions was the naming convention of the Astlib.

For example, some things are named like Ptyp_any, and others are named like ptype_name. Without explanations, I can't really tell what the consideration is for making "typ" and "type" different. Another example is naming like pcd_name, and it is hard for me to tell what does pcd stands for. If we can add some explanations for the naming convention of Ast_lib, it will definitely be helpful on the code quality of ppx extension developed by beginners like me, as they can make sure the naming in their extension is aligned to the standard (both ppxlib and the OCaml AST).