ansible-output subcommand: refactor; add post-processor support (#402)

* Improve validation.

* Refactoring.

* Update changelog fragment.

* Prevent error reporting.

* Add post-processor support.

* Improve docs.

* Fix indentation, add link.

Author: felixfontein

changelogs/fragments/397-ansible-output.yml

@@ -1,3 +1,5 @@
 minor_changes:
   - "Add a new subcommand ``ansible-output`` which allows to render Ansible output into RST code blocks
-     (https://github.com/ansible-community/antsibull-docs/pull/397, https://github.com/ansible-community/antsibull-docs/pull/401)."
+     (https://github.com/ansible-community/antsibull-docs/pull/397,
+     https://github.com/ansible-community/antsibull-docs/pull/401,
+     https://github.com/ansible-community/antsibull-docs/pull/402)."

docs/ansible-output.md

@@ -91,6 +91,9 @@ Also take a look at the example further below which demonstrates all of them.
 
 * The `prepend_lines` key allows to prepend a multi-line YAML string to the `ansible-playbook` output.
 
+* The `postprocessors` key allows to define a list of post-processors.
+  This is explained in more detail in the [Post-processing ansible-playbook output section](#post-processing-ansible-playbook-output).
+
 An example looks like this. The `console` code block contains the generated result:
 ```rst
 This is an Ansible task we're going to reference in the playbook:
@@ -146,6 +149,9 @@ This is an Ansible task we're going to reference in the playbook:
         previous_code_block: yaml+jinja
         previous_code_block_index: -1
 
+    # No post-processors
+    postprocessors: []
+
     # The actual playbook to run:
     playbook: |-
       @{# Use the 'hosts' variable defined above #}@
@@ -177,6 +183,131 @@ The task produces the following output:
     }
 ```
 
+## Post-processing ansible-playbook output
+
+Out of the box, you can post-process the `ansible-playbook` output in some ways:
+
+* Skip a fixed number of lines at the top (`skip_first_lines`) or bottom (`skip_last_lines`).
+* Prepend lines to the output (`prepend_lines`).
+
+This, together with chosing an appropriate callback plugin
+(like [community.general.tasks_only](https://docs.ansible.com/ansible/devel/collections/community/general/tasks_only_callback.html))
+gives you a lot of freedom to get the output you want.
+
+In some cases, it is not sufficient though.
+For example, if you want to extract YAML output, and present it in a way that [matches your yamllint configuration](https://ansible.readthedocs.io/projects/antsibull-nox/config-file/#yamllint-part-of-the-yamllint-session).
+The default callback's YAML output suffers from [PyYAML's list indentation issue](https://github.com/yaml/pyyaml/issues/234),
+which causes problems with many yamllint configurations.
+Also, the [ansible.builtin.default callback's YAML output](https://docs.ansible.com/ansible/devel/collections/ansible/builtin/default_callback.html#parameter-result_format) is indented by 4 spaces,
+while most YAML is expected to be indented by 2 spaces.
+
+If you use the above settings (`skip_first_lines` / `skip_last_lines`) to extract only the YAML content of one task of the playbook's output,
+you can for example use [Pretty YAML (pyaml)](https://pypi.org/project/pyaml/) to reformat it.
+For that, you can use the `postprocessors` list to specify a post-processor command:
+```yaml
+postprocessors:
+  - command:
+      - python
+      - "-m"
+      - pyaml
+```
+This tells `antsibull-docs ansible-output` to feed the extracted output
+(with `skip_first_lines`, `skip_last_lines`, and `prepend_lines` already processed)
+through standard input into the `python -m pyaml` process,
+and use its output instead.
+
+A full example looks like this:
+```rst
+.. code-block:: yaml+jinja
+
+   input:
+     - k0_x0: A0
+       k1_x1: B0
+       k2_x2: [C0]
+       k3_x3: foo
+     - k0_x0: A1
+       k1_x1: B1
+       k2_x2: [C1]
+       k3_x3: bar
+
+   target:
+     - {after: a0, before: k0_x0}
+     - {after: a1, before: k1_x1}
+
+   result: "{{ input | community.general.replace_keys(target=target) }}"
+
+.. ansible-output-data::
+
+    env:
+      ANSIBLE_CALLBACK_RESULT_FORMAT: yaml
+    variables:
+      data:
+        previous_code_block: yaml+jinja
+    postprocessors:
+      - command:
+          - python
+          - "-m"
+          - pyaml
+    language: yaml
+    skip_first_lines: 4
+    skip_last_lines: 3
+    playbook: |-
+      - hosts: localhost
+        gather_facts: false
+        tasks:
+          - vars:
+              @{{ data | indent(8) }}@
+            ansible.builtin.debug:
+              var: result
+
+This results in:
+
+.. code-block:: yaml
+
+   result:
+     - a0: A0
+       a1: B0
+       k2_x2:
+         - C0
+       k3_x3: foo
+     - a0: A1
+       a1: B1
+       k2_x2:
+         - C1
+       k3_x3: bar
+```
+
+Right now there are two kind of post-processor entries in `postprocessors`:
+
+1. Command-based post-processors:
+
+    You can provide a list `command`.
+    This command is executed,
+    the input fed in through standard input,
+    and its standard output is taken as the output.
+
+    Example:
+    ```yaml
+    postprocessors:
+      - command:
+          - python
+          - "-m"
+          - pyaml
+    ```
+
+2. Name-reference post-processors:
+
+    You can use `name` to reference a named globally defined post-processor.
+    This is right now only possible in collections,
+    since you need to define these in the collection's config file
+    (`docs/docsite/config.yml` - see the [Collection usage section](#collection-usage)).
+
+    Example:
+    ```yaml
+    postprocessors:
+      - name: reformat-yaml
+    ```
+
 ## Standalone usage
 
 If you want to update a RST file, or all RST files in a directory, you can run antsibull-docs as follows:
@@ -196,7 +327,8 @@ If you run `antsibull-docs ansible-output` without a path, it assumes that you a
 It will check all `.rst` files in `docs/docsite/rst/`, if that directory exists,
 and load configuration from `docs/docsite/config.yml`.
 (See [more information on that configuration file](../collection-docs/#configuring-the-docsite).)
-The configuration allows you to specify entries for `env` for all code blocks:
+The configuration allows you to specify entries for `env` for all code blocks,
+and you can define global post-processors that can be referenced in `postprocessors`:
 
 ```yaml
 ---
@@ -206,9 +338,21 @@ ansible_output:
   global_env:
     ANSIBLE_STDOUT_CALLBACK: community.general.tasks_only
     ANSIBLE_COLLECTIONS_TASKS_ONLY_NUMBER_OF_COLUMNS: 80
+
+  # Global post-processors for Ansible output
+  global_postprocessors:
+    # Keys are the names that can be referenced in ansible-output-data directives
+    reformat-yaml:
+      # For CLI tools, you can specify a command that accepts input on stdin
+      # and outputs the result on stdout:
+      command:
+        - python
+        - "-m"
+        - pyaml
 ```
 
-This is useful to standardize the callback and its settings for most code blocks in a collection's extra docs.
+This is useful to standardize the callback and its settings for most code blocks in a collection's extra docs,
+and set up a pre-defined set of post-processors that can be used everywhere.
 
 ## Usage in CI
 

src/antsibull_docs/cli/doc_commands/ansible_output.py

@@ -32,7 +32,12 @@
 from sphinx_antsibull_ext.directive_helper import YAMLDirective
 from sphinx_antsibull_ext.schemas.ansible_output_data import (
     AnsibleOutputData,
+    Postprocessor,
+    PostprocessorCLI,
+    PostprocessorNameRef,
     VariableSource,
+    VariableSourceCodeBlock,
+    VariableSourceValue,
 )
 
 from ... import app_context
@@ -163,14 +168,15 @@ def _find_blocks(
 @dataclass
 class Environment:
     env: dict[str, str]
+    global_postprocessors: dict[str, Postprocessor]
 
 
 def _get_variable_value(
     *, key: str, value: VariableSource, previous_blocks: list[CodeBlockInfo]
 ) -> str:
-    if value.value is not None:
+    if isinstance(value, VariableSourceValue):
         return value.value
-    if value.previous_code_block is None:
+    if not isinstance(value, VariableSourceCodeBlock):
         raise AssertionError(  # pragma: no cover
             "Implementation error: cannot handle {value!r}"
         )
@@ -240,6 +246,70 @@ def _strip_common_indent(lines: list[str]) -> list[str]:
     return [line[indent:] for line in lines]
 
 
+def _massage_stdout(
+    stdout: str,
+    *,
+    skip_first_lines: int = 0,
+    skip_last_lines: int = 0,
+    prepend_lines: str | None = None,
+) -> list[str]:
+    # Compute result lines
+    lines = [line.rstrip() for line in stdout.split("\n")]
+    lines = _strip_empty_lines(lines)
+
+    # Skip lines
+    if skip_first_lines > 0:
+        lines = lines[skip_first_lines:]
+    if skip_last_lines > 0:
+        lines = lines[:-skip_last_lines]
+
+    # Prepend lines
+    if prepend_lines:
+        lines = prepend_lines.split("\n") + lines
+
+    return _strip_common_indent(_strip_empty_lines(lines))
+
+
+def _apply_postprocessor(
+    lines: list[str],
+    *,
+    cwd: Path,
+    env: dict[str, str],
+    postprocessor: Postprocessor,
+    environment: Environment,
+) -> list[str]:
+    flog = mlog.fields(func="_apply_postprocessor")
+
+    if isinstance(postprocessor, PostprocessorNameRef):
+        ref = postprocessor.name
+        try:
+            postprocessor = environment.global_postprocessors[ref]
+        except KeyError:
+            raise ValueError(  # pylint: disable=raise-missing-from
+                f"No global postprocessor of name {ref!r} defined"
+            )
+
+    if isinstance(postprocessor, PostprocessorCLI):
+        flog.notice("Run postprocessor command: {}", postprocessor.command)
+        try:
+            result = subprocess.run(
+                postprocessor.command,
+                capture_output=True,
+                input="\n".join(lines) + "\n",
+                cwd=cwd,
+                env=env,
+                check=True,
+                encoding="utf-8",
+            )
+        except subprocess.CalledProcessError as exc:
+            raise ValueError(
+                f"{exc}\nError output:\n{exc.stderr}\n\nStandard output:\n{exc.stdout}"
+            ) from exc
+        lines = _massage_stdout(result.stdout)
+
+    return lines
+
+
 def _compute_code_block_content(
     data: _AnsibleOutputDataExt,
     *,
@@ -277,22 +347,27 @@ def _compute_code_block_content(
             ) from exc
 
         flog.notice("Post-process result")
-
-        # Compute result lines
-        lines = [line.rstrip() for line in result.stdout.split("\n")]
-        lines = _strip_empty_lines(lines)
-
-        # Skip lines
-        if data.data.skip_first_lines > 0:
-            lines = lines[data.data.skip_first_lines :]
-        if data.data.skip_last_lines > 0:
-            lines = lines[: -data.data.skip_last_lines]
-
-        # Prepend lines
-        prepend_lines = (
-            data.data.prepend_lines.split("\n") if data.data.prepend_lines else []
+        lines = _massage_stdout(
+            result.stdout,
+            skip_first_lines=data.data.skip_first_lines,
+            skip_last_lines=data.data.skip_last_lines,
+            prepend_lines=data.data.prepend_lines,
         )
-        return _strip_common_indent(_strip_empty_lines(prepend_lines + lines))
+        for postprocessor in data.data.postprocessors:
+            flog.notice("Run post-processor {}", postprocessor)
+            try:
+                lines = _apply_postprocessor(
+                    lines,
+                    cwd=directory,
+                    env=env,
+                    postprocessor=postprocessor,
+                    environment=environment,
+                )
+            except ValueError as exc:
+                raise ValueError(
+                    f"Error while running post-processor {postprocessor}:\n{exc}"
+                ) from exc
+        return lines
 
 
 def _replace(
@@ -581,10 +656,12 @@ def get_environment(
         else:
             collections_path = f"{collection_path}"
         env["ANSIBLE_COLLECTIONS_PATH"] = collections_path
+    postprocessors = {}
     if collection_config is not None:
         env.update(collection_config.ansible_output.global_env)
+        postprocessors.update(collection_config.ansible_output.global_postprocessors)
     flog.notice("Environment template: {}", env)
-    return Environment(env=env)
+    return Environment(env=env, global_postprocessors=postprocessors)
 
 
 def detect_color(*, force: bool | None = None) -> bool:

src/antsibull_docs/schemas/collection_config.py

@@ -5,8 +5,23 @@
 # SPDX-FileCopyrightText: 2023, Ansible Project
 """Schemas for collection config files."""
 
+import typing as t
+
 import pydantic as p
 
+from sphinx_antsibull_ext.schemas.ansible_output_data import (
+    Postprocessor,
+    PostprocessorNameRef,
+)
+
+
+def _is_not_name_ref(value: Postprocessor) -> Postprocessor:
+    if isinstance(value, PostprocessorNameRef):
+        raise ValueError(
+            "Cannot define name reference postprocessors in collection config"
+        )
+    return value
+
 
 class ChangelogConfig(p.BaseModel):
     # Whether to write the changelog
@@ -17,6 +32,11 @@ class AnsibleOutputConfig(p.BaseModel):
     # Environment variables to inject for every ansible-output-data
     global_env: dict[str, str] = {}
 
+    # Named postprocessors
+    global_postprocessors: dict[
+        str, t.Annotated[Postprocessor, p.AfterValidator(_is_not_name_ref)]
+    ] = {}
+
     @p.field_validator("global_env", mode="before")
     @classmethod
     def convert_dict_values(cls, obj):

src/sphinx_antsibull_ext/directives.py

@@ -76,6 +76,11 @@ class _AnsibleOutputDataDirective(YAMLDirective[AnsibleOutputData]):
     wrap_as_data = True
     schema = AnsibleOutputData
 
+    def _handle_error(self, message: str, from_exc: Exception) -> list[nodes.Node]:
+        # Do not report errors when simply building docs.
+        # Errors should be reported when running 'antsibull-docs lint-collection-docs'.
+        return []
+
     def _run(self, content_str: str, content: AnsibleOutputData) -> list[nodes.Node]:
         # This directive should produce no output. It is used in the ansible-output subcommand.
         return []