Skip to content

Add historical journald and log export flags to operator debug command#26410

Merged
tehut merged 10 commits intomainfrom
f-NMD-141-debug-export
Aug 4, 2025
Merged

Add historical journald and log export flags to operator debug command#26410
tehut merged 10 commits intomainfrom
f-NMD-141-debug-export

Conversation

@tehut
Copy link
Copy Markdown
Contributor

@tehut tehut commented Aug 1, 2025

Description

Support requested the ability to retrieve historical journald logs as part of the debug capture returned by nomad operator debug. PR #26178 introduced a MonitorExport() RPC capable of streaming the contents of an agent's log file (if file logging is enabled) or its journald logs.

This PR adds that functionality to operator debug with the inclusion of either a log-file-export=true flag or a -log-lookback=<duration> flag. An agent's historical log capture runs alongside the capture of running logs and results in both monitor.log and monitor_export.log files appearing in each agent's debug capture.

Note to reviewers: I exported a test helper from PR #26178 for this new test case. The helper generates a golden log file for testing against stream results. And since the helper is now exported, I also refactored a few tests from that PR to use it as well since they were using the same mechanism.

Testing & Reproduction steps

To run locally:

  1. Build branch
  2. Run a debug command with one of the historical log operators e.g nomad operator debug -duration=10s -interval=5s -log-file-export=true
  3. Unzip the debug archive to display a monitor.log and monitor_export.log file for each agent in the cluster.
    

Example below

Screenshot 2025-08-01 at 4 35 25 PM

Links

Requested in: NMD-141

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.

@tehut tehut added theme/docs Documentation issues and enhancements theme/cli type/enhancement backport/1.10.x labels Aug 1, 2025
@tehut tehut force-pushed the f-NMD-141-debug-export branch from a04fe42 to b974527 Compare August 2, 2025 00:00
@tehut tehut force-pushed the f-NMD-141-debug-export branch from b974527 to 5596702 Compare August 2, 2025 00:15
@tehut tehut force-pushed the f-NMD-141-debug-export branch from 0f00e89 to 7769506 Compare August 2, 2025 00:47
@tehut tehut marked this pull request as ready for review August 2, 2025 00:47
@tehut tehut requested review from a team as code owners August 2, 2025 00:47
@tehut tehut changed the title Add -log-file-export and -log-lookback commands to add historical log to Add historical journald and log export flags to operator debug command Aug 2, 2025
@tehut tehut force-pushed the f-NMD-141-debug-export branch from 13171be to 3e606d4 Compare August 2, 2025 02:11
Comment thread website/content/commands/operator/debug.mdx Outdated
Comment thread command/operator_debug.go Outdated
Comment thread command/operator_debug.go Outdated
@tehut tehut force-pushed the f-NMD-141-debug-export branch from 13762cc to 0c4082f Compare August 4, 2025 16:40
Copy link
Copy Markdown
Contributor

@aimeeu aimeeu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the docs! I left a few style guide suggestions.

- `log-file-export`: Include agents' Nomad logfiles in the debug capture.
The historical log export monitor runs concurrently with the log monitor
and ignores the `-log-level` and `-log-include-location` flags used to
configure that monitor. Nomad will return an error if the agent does not
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
configure that monitor. Nomad will return an error if the agent does not
configure that monitor. Nomad returns an error if the agent does not

- `log-lookback`: Include historical journald logs in the debug capture. The
journald export monitor runs concurrently with the log monitor and ignores
the `-log-level` and `-log-include-location` flags passed to that monitor.
This flag is only available on Linux systems, see the `-log-file-export`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This flag is only available on Linux systems, see the `-log-file-export`
This flag is only available on Linux systems. Refer to the `-log-file-export`

@tehut tehut force-pushed the f-NMD-141-debug-export branch from d8128a5 to 0b52572 Compare August 4, 2025 20:13
Copy link
Copy Markdown
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tehut tehut merged commit 21841d3 into main Aug 4, 2025
39 checks passed
@tehut tehut deleted the f-NMD-141-debug-export branch August 4, 2025 20:55
tehut added a commit that referenced this pull request Aug 4, 2025
#26410)

* Add -log-file-export and -log-lookback commands to add historical log to
debug capture
* use monitor.PrepFile() helper for other historical log tests
tehut added a commit that referenced this pull request Aug 5, 2025
#26410) (#26427)

* Add -log-file-export and -log-lookback commands to add historical log to
debug capture
* use monitor.PrepFile() helper for other historical log tests

Co-authored-by: tehut <tehut.m.g@gmail.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Dec 3, 2025

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Dec 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

theme/cli theme/docs Documentation issues and enhancements type/enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants