[strawman]: declarative "template tools"

[jonasfj]

There is some debate about making first-class dartdoc directives for tools like snippets, and I was thinking why don't we have "template tools", or rather tools that don't involve running dart code, but maybe just evaluating a template.

This is an extremely ugly attempt at defining what that could look like, without us inventing a new language and still giving template authors some power.

Warning

Please don't think of this as a serious proposal, it's more of an exploratory idea 🤣

dartdoc:
  # What if we allowed "template tools"?
  template_tools:
    # We could almost define something like snippets that way.
    snippets:
      # Snippets don't have flags or options, so this is just illustrative
      flags:
        - warning     # ArgParser.addFlag('warning')      , allowing --[no]-warning (defaulting to false)
      options:
        - language    # ArgParser.addOption('language')   , allowing --language=foo

      # We have a context dictionary as follows:
      #   source.line
      #   source.column
      #   source.path
      #   package.name
      #   element.name
      #   invocationIndex
      #   input                         # text between {@tool snippets} and {@end-tool}
      #   arguments.raw                 # space separated arguments given in {@tool} invocation
      #   arguments.warning             # true, if --warning was given in {@tool} invocation
      #   arguments.language            # value, if --language=value was given in {@tool} invocation
      #
      # Using this context we can do some limited steps like:
      #  - extract variables from context using regular expressions and save them to context again.
      #  - assert values from context and throw an error instead of rendering a faulty template!
      steps:
      # Create a new "description" field in the context dictionary, running pattern as regex against text
      - extract: description
        text:           '{{input}}'
        pattern:        '^(?<content>.*)^```'
        multiLine:      true
        caseSensitive:  true
        dotAll:         true
      - extract: code
        text:           '{{input}}'
        pattern:        '^```(?<language>.*)\n(?<source>.*)^```'
        multiLine:      true
        caseSensitive:  true
        dotAll:         true
      # Evaluate assertion with package:boolean_selector (already used in package:test)
      # Giving template tool authors an option to sanity check data before the template is rendered.
      - assert: 'code.hasMatch && description.hasMatch'
        message: 'Failed to extract code and language from snippet'

      # After running "steps" we've expanded context with:
      #   description.hasMatch  # True, if regex was matched
      #   description.content   # capture group content
      #   code.hasMatch         # True, if regex was matched!
      #   code.language         # capture group language
      #   code.source           # capture group source
      #
      # We can access context in the template using mustache as illustrated below:
      template: |
        {@inject-html}
        <a name="{{element.name}}.{{invocationIndex}}"></a>
        <div class="snippet snippet-container anchor-container">
          {{description}}
          <a class="anchor-button-overlay anchor-button" title="Copy link to clipboard"
            onmouseenter="fixHref(this, '{{element.name}}.{{invocationIndex}}');" onclick="fixHref(this, '{{element.name}}.{{invocationIndex}}'); copyStringToClipboard(this.href);"
            href="#">
            <i class="material-icons anchor-image">link</i>
          </a>
          <div class="copyable-container">
            <button class="copy-button-overlay copy-button" title="Copy to clipboard"
              onclick="copyTextToClipboard(findSiblingWithId(this, 'sample-code'));">
              <i class="material-icons copy-image">content_copy</i>
            </button>
            <pre class="language-{{code.language}}" id="sample-code"><code class="language-{{code.language}}">{{code.source}}</code></pre>
          </div>
        </div>
        {@end-inject-html}

Note that context can probably just be Map<String, String>, we dont' have to treat foo.bar as sub-property access :D

Note:

• You can still do some attacks by using degenerate regular expressions
  • We could probably accept that since it'll just loop indefinitely (or be slow), or,
  • We could evaluate the regular expressions in an isolate (which is cheap).
• Regexes gives template tool authors a lot of power to do at-least something.
• The steps of repeated regex evaluation is a powerful, but also horrific programming environment 🤣

---

I'm not sure this is the way, we could also implement a small expression language that can be safely evaluated in templates. Options range form jinja-like things, to implementing a safe subset Dart in a small interpreter. To porting json-e to Dart and feeding the output into mustache.

But we could probably get far with regexes, boolean_selector for assertions and mustache for templating.

---

All this being explored: We should perhaps ask the question if we want people inventing their own HTML section types. Or whether it'd be better to argue that:

• (A) users should be using existing markdown constructs (there aren't much missing).
• (B) We should introduce dartdoc directives when good markdown constructs are missing.

In the case of snippets, it could be argued that:

• Using a ## Section title in markdown is better than a snippet macro.
• All sections in dartdoc markdown should have a perma-link.
• All code blocks should have a "copy to clipboard" button.

Though, we'd have to give up the special background color for snippet section blocks.

[jonasfj]

Iteration 2

Okay, I've tried to simplify it a bit, and not make it as complex, in this case you'd define "template tools" as follows, we call them just "templates" below:

dartdoc:
  templates:
    snippet:
      arguments:
        key1: default_value1
        key2: default_value2
      extract:
        my_code:
          pattern:        '^```(?<language>.*)\n(?<source>.*)^```'
          caseSensitive:  true  # optional, default true
          multiLine:      true  # optional, default false
          dotAll:         true  # optional, default false
      # Context available in template:
      #   source.line
      #   source.column
      #   source.path
      #   package.name
      #   element.name
      #   invocationIndex
      #   body                        # value between open/close tags in {@expand snippet}body{/expand}
      #   my_code[0].language         # Named group from first match of extract.my_code.pattern against body
      #   my_code[0].source
      #   key1                        # {@expand snippet key1=foo key2=bar}
      #   key2
      template: |
        Mustache template with access to context, like: {{package.name}}.

They would be invoked with syntax like:

Invocation in block mode:

/// {@expand <templateName> [key="<value>" ...]}
/// <body>
/// {/expand}

Invocation inline mode:

/// {@expand <templateName> [key="<value>" ...] /}

Example

The following dartdoc_options.yaml defines sample and jsonTypesOnly:

dartdoc:
  templates:
    # Template that wraps around a body and accepts some arguments
    # This could replace some existing {@tool} usages.
    sample:
      arguments:
        file: # no default value
      template: |
        {@inject-html}<div class="blue-box">{@end-inject-html}
        {{body}}
        
        Try out this sample locally using:
        `flutter create --sample=package:{{package.name}}/../{{file}}`
        
        {$example {{file}}
        {@inject-html}</div>{@end-inject-html}
    
    # Easy way to insert a sentence that is often repeated
    # This could replace {@template} and {@macro} directives
    jsonTypesOnly:
      arguments:
        arg: # no default value
      template: | # {{#arg}} and {{^arg} is "if" and "ifnot" in mustache
        {{#arg}}
        The [{{arg}}] parameter for {{element.name}} has type [Object?],
        [{/arg}}
        {{^arg}]
        {{element.name}} accepts [Object?],
        {{/arg}}
        but only supports JSON types:
         * Map,
         * List,
         * String,
         * num,
         * bool, and,
         * null

Example usage

/// {@expand sample file="path/to/file.dart}
/// bla bla bla..
/// {/expand}
///
/// {@expand jsonTypesOnly arg=value}
void encodeAsJson(Object? value) {

Would become:

/// {@inject-html}<div class="blue-box">{@end-inject-html}
/// bla bla bla..
///
/// Try out this sample locally using:
/// `flutter create --sample=package:my_package/../path/to/file.dart`
///
/// {$example path/to/file.dart}
/// {@inject-html}</div>{@end-inject-html}
///
/// The [value] parameter for encodeAsJson has type [Object?],
/// but only supports JSON types:
///  * Map,
///  * List,
///  * String,
///  * num,
///  * bool, and,
///  * null
void encodeAsJson(Object? value) {

---

I don't know if:

• {@expand} is the right name for the directive (but macro is taken).
• dartdoc_options.yaml is the best place to write these, perhaps something like doc/templates/<templateName>.md with YAML frontmatter would be better.
• I don't know if it would better to support two kinds of "template tools":
  • One that has a body (and requires both open and close tags)
  • One that doesn't have a body (and doesn't require a close tag)
• I don't know if extracting values from body using regular expressions is even necessary.
  • Allowing does give "template tool" authors a lot of power.
• I don't love mustache, but it's easy to implement a small subset, well-defined, dumb, and has lots of tutorials.

I do like that:

• We can deprecate {@template} and {@macro} directives.
• The mechanism can take parameters, I missed this in {@template} and {@macro} directives.
• That templates are stored in a single location not scattered across the code.
• That we have open/close tags {@expand}{/expand} and self-closing tags {@expand/}

[sigurdm]

dartdoc_options.yaml is the best place to write these, perhaps something like doc/templates/.md with YAML frontmatter would be better.

IMO yes, much better to declare in their own files. These are not "options".

I don't know if extracting values from body using regular expressions is even necessary.

It allows for more "fluent" looking raw dartdoc comments. But I kinda agree that just putting yaml in the body would get us really far, and be a much less frustrating user experience (who likes debugging non-matching regexps?) Perhaps writing up competing examples for our known use-cases would help us decide if we need it.