Skip to content

Conversation

chrisroberts
Copy link
Member

@chrisroberts chrisroberts commented Jun 9, 2025

Description

When performing a graceful shutdown a channel is used to wait for
the agent to leave. The channel is closed when the agent leaves
successfully, but it also is closed within a deferral. If the
agent successfully leaves and closes the channel, a panic will
occur when the channel is closed the second time within the
deferral. To prevent this from occurring, the channel closing
is wrapped within a OnceFunc so the channel is only closed
once.

Testing & Reproduction steps

Panic encountered during shutdown:

panic: close of closed channel
goroutine 1 [running]:
github.com/hashicorp/nomad/command/agent.(*Command).terminateGracefully(0xc000948720, 0xc000538e70, {0x0?, 0x0?})
        github.com/hashicorp/nomad/command/agent/command.go:1019 +0x2a5
github.com/hashicorp/nomad/command/agent.(*Command).handleSignals(0xc000948720)
        github.com/hashicorp/nomad/command/agent/command.go:1069 +0x811
github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc000948720, {0xc000072120, 0x2, 0x2})
        github.com/hashicorp/nomad/command/agent/command.go:909 +0xf25
github.com/hashicorp/cli.(*CLI).Run(0xc000a7cb40)
        github.com/hashicorp/[email protected]/cli.go:265 +0x4de
main.Run({0xc000072110, 0x3, 0x3})
        github.com/hashicorp/nomad/main.go:111 +0x225
main.main()
        github.com/hashicorp/nomad/main.go:82 +0x45

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.

tgross
tgross previously approved these changes Jun 10, 2025
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

When performing a graceful shutdown a channel is used to wait for
the agent to leave. The channel is closed when the agent leaves
successfully, but it also is closed within a deferral. If the
agent successfully leaves and closes the channel, a panic will
occur when the channel is closed the second time within the
deferral. To prevent this from occurring, the channel closing
is wrapped within a `OnceFunc` so the channel is only closed
once.
@chrisroberts chrisroberts requested a review from tgross June 12, 2025 15:56
@chrisroberts chrisroberts added backport/1.10.x backport to 1.10.x release line backport/ent/1.8.x+ent Changes are backported to 1.8.x+ent backport/ent/1.9.x+ent Changes are backported to 1.9.x+ent labels Jun 12, 2025
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@chrisroberts chrisroberts merged commit 4dbf645 into main Jun 12, 2025
46 checks passed
@chrisroberts chrisroberts deleted the b-shutdown-panic branch June 12, 2025 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/ent/1.8.x+ent Changes are backported to 1.8.x+ent backport/ent/1.9.x+ent Changes are backported to 1.9.x+ent backport/1.10.x backport to 1.10.x release line
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants