Add nomad monitor export command

[tehut]

Description

The nomad monitor export command introduces the ability for nomad to export logs a given agent has written to journald or to the nomad log file. Journald logs can be requested for a specific period of time while we just return the agent's entire nomad log file. Introducing this RPC is a prerequisite for adding journald logs to the nomad support bundle.

Pasting a link to a comment deep in the thread regarding fixes for inconsistencies related to ending the stream.
#26178 (comment)

Testing & Reproduction steps

Links

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Pulled this from the Contributor checklist for new cli commands, I struck out a few items that didn't seem to fit this use case but I'm happy to revisit them if I was wrong there.

CLI Command Checklist

• Consider similar commands in Consul, Vault, and other tools. Is there
  prior art we should match? Arguments, flags, env vars, etc?
• New file in command/ or in an existing file if a subcommand
• Test new command in command/ package
• Implement autocomplete
• Update help text
• Register new command in command/commands.go
• Implement and test new HTTP endpoint in command/agent/<command>_endpoint.go
• Register new URL paths in command/agent/http.go
• Implement and test new RPC endpoint in nomad/<command>_endpoint.go
• Implement and test new Client RPC endpoint in
  client/<command>_endpoint.go (For client endpoints like Filesystem only)
• Implement and test new nomad/structs/ package Request and Response structs (cstructs, but still)
• Implement and test new api/ package helper methods
  I added a monitorHelper function to more cleanly share the existing monitor code in /api/agent.go. I did not add an additional test for the helper or the MonitorExport command. Mostly because I'd have to update the Client Config structs to be able to set the LogFile value on the api Client. But the only difference between the Monitor & MonitorExport commands in api/agent.go is which path is passed to the monitorHelper and helper is exercised in the existing monitor tests.

* [ ] For nested commands make sure all intermediary subcommands exist (for
example, nomad acl, nomad acl policy, and nomad acl policy apply must
all be valid commands)
* [ ] If the command has a status subcommand consider adding a search context
in nomad/search_endpoint.go and update command/status.go
* [ ] Implement -json (returns raw API response)
* [ ] Implement -t (format API response using gotemplate)
* [ ] Implement -verbose (expands truncated UUIDs, adds other detail)
* [ ] Implement and test new api/ package Request and Response structs

Docs

• Changelog
• API docs https://developer.hashicorp.com/nomad/api
• CLI docs https://developer.hashicorp.com/nomad/docs/commands
• If adding new docs see website README
• Consider if it needs a guide https://developer.hashicorp.com/nomad/guides/index.html

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

[tgross]

I know you're not quite done @tehut but I made a first pass over this.

[tgross]

This is looking good @tehut! I've also checked this out locally and ran it against a multinode cluster to check things like goroutine leaks, truncated reads, etc. and it looks great!

I've left a little pile of comments and questions but most of it is small stuff at this point.

[tgross]

Looks great! I've left a small number of suggestions and one last-triple-check question, but once those are resolved we can squash this and merge it!

[tgross]

This is also unused, looks like. The callers are in other packages, so they can't see this field anyways. But we're not using it in test either.

[tehut]

I'm not super committed to keeping this in but I wanted to try it as an exported field in light of the point you raised yesterday about the StreamFramer and StreamReader also having a configurable window size. I like how it's currently configured where the producer is pushing 1/2 as much data as each consumer is ready to process but it seems like a dial folks might want to use down the road? Or does that just introduce another thing to keep track of/test/validate?

[tgross]

Works for me. 👍

This whole project really points to a need to revisit some of the plumbing for how uni-directional and bi-directional streaming is designed, as it was super painful to add a new endpoint. So having configuration knobs handy will be useful later if we ever find the time to set aside for that.

[tgross]

LGTM!