Support for Logstash plugins (WIP)

[karenzone]

WIP - Add content from Slack convo with @Mpdreamz and @karenzone (with some bonus content for context)

Topic: Questions and concerns about migrating Logstash plugin docs

• Plugin doc source files. Logstash plugin docs live in 200+ repos in the logstash-plugins github org--not the elastic github org. (This causes problems with github automation.)

• Validation. There's no Buildkite validation or linting on Logstash plugin doc source files. Problems don't manifest until after a docs PR has been merged in the plugin repo, the gem published to rubygems.org, the version tag pushed to the repo, the plugin docs generated (for either the Logstash Reference[1] or the Logstash Versioned Plugin Reference[2]), and changes staged in a PR[3] in the logstash-docs repo.

  • When the PR is created in logstash-docs, BuildKite validation checks it. Fortunately, we know not to merge a PR that doesn't pass CI. However, the bad docs are tied to the plugin version tag, and we have to make the fix in the source file, bump the version, publish to rubygems.org again, and make fixes in generated content that's already out there.

• Plugin doc source files are templates[4]. That is, they are not fully functional docs. We have docs tooling[5][6] that populates variables and does some transformation so that we have different output from the same content for the Logstash Reference and the Versioned Plugin Reference.

• Plugin doc headers.

  • @Mpdreamz: "In docs V3 we don't support referencing files outside the current doc set.
    e.g include::{include_path}/plugin_header.asciidoc[] which resolves to ../../../../logstash/docs/include/plugin_header.asciidoc"
  • @karenzone: This was raised as an issue in a previous migration discussion (for docsmobile maybe?) and we determined that this approach would be permissible. Sounds like we need to revisit and rethink possible workaround.
  • Spitballing here, but we might be able to work around this issue if we make the stack versioned plugins that are currently included in the Logstash Reference a stand-alone doc built out of the logstash-docs repo. But that approach is not without its own complexities that we'll need to consider.

• Logstash plugins in IA.

  • @Mpdreamz: "i believe the versioned plugin reference goes away we'll only have to content once (assuming under the logstash reference) as part of IA?"
  • @karenzone: I don't think that's been decided yet. LSR and VPR serve different purposes. Slack threads: https://elastic.slack.com/archives/C05PP2LEC1X/p1736869794964169 and https://elastic.slack.com/archives/C05PP2LEC1X/p1736871825107279

• Impact of versioning changes.

  • Question from @lcawl (paraphrased): "What's the impact of the new versioning strategy on stack-versioned plugin presentation?"
  • @karenzone: TBD. Seems like users will always need to know which plugin versions are compatible with a Logstash core version. (Note that matching plugin/stack versions is a rough approximation controlled by the Gemfile.jruby-3.1.lock.release file that we generate at feature freeze.)

References

[1] Logstash Reference (LSR).
"I'm running Logstash 8.17. What plugin versions are compatible with this version of Logstash? What versions of default plugins are installed with Logstash by default?"
[2] Logstash Versioned Plugin Reference.
"I'm using logstash-output-elasticsearch v11.22.8. What config options are available to me?"
[3] Sample logstash-docs PRs:

• Plugin files for Logstash Reference (LSR):
  • updated docs for main logstash-docs#1771
  • updated docs for 8.17 logstash-docs#1787
  • updated docs for 8.16 logstash-docs#1786
• Plugin files Versioned Plugin Reference (VPR):
  • auto generated update of versioned plugin documentation logstash-docs#1788

[4] Sample plugin source doc: https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/main/docs/index.asciidoc?plain=1

• Generated output for LSR: https://github.com/elastic/logstash-docs/blob/main/docs/plugins/outputs/elasticsearch.asciidoc?plain=1
• Generated output for VPR: https://github.com/elastic/logstash-docs/blob/versioned_plugin_docs/docs/versioned-plugins/outputs/elasticsearch-v11.22.10.asciidoc?plain=1

[5] Logstash Reference (LSR) docgen: https://github.com/elastic/docs-tools/blob/master/plugindocs.rb
[6] Versioned Plugin Reference (VPR) docgen: https://github.com/elastic/docs-tools/blob/master/versioned_plugins.rb

Previous considerations and related issues

• Doc: Create plan to convert plugin docs source and output from ASCIIDOC to MD (and update tooling) logstash#16852
• https://github.com/elastic/ingest-dev/issues/4881
• https://github.com/elastic/platform-docs-team/issues/283
• https://github.com/elastic/platform-docs-team/issues/409
• https://github.com/elastic/platform-docs-team/issues/284

[karenzone]

cc:/ @robbavey

[karenzone]

@karenzone Note to self: https://github.com/elastic/logstash-filter-elastic_integration docs are also an outlier from a docgen PoV.