Skip to content

Conversation

allisonlarson
Copy link
Member

@allisonlarson allisonlarson commented Jul 7, 2025

Description

Adds the node_pool to the blockedEval metrics that get emitted for
resource/cpu, along with the dc and node class.
Docs have been updated to include the new metric label

Testing & Reproduction steps

The node_pool has been added to the automated tests, and seen when using manual tests and observing metrics emitted.

Links

Fixes #25933

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.

@aimeeu aimeeu added the theme/docs Documentation issues and enhancements label Jul 8, 2025
@aimeeu
Copy link
Contributor

aimeeu commented Jul 9, 2025

@allisonlarson The website/content/docs/operations/metrics-reference.mdx is now website/content/docs/reference/metrics.mdx
Please let me know if you want me to resolve the conflict and push a commit.

Adds the node_pool to the blockedEval metrics that get emitted for
resource/cpu, along with the dc and node class.
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach looks good, but I think we're missing a few spots where we need to add the pool. I did rg -e '\&structs\.Evaluation' and it looks like we're missing 3 spots in nomad/job_endpoint.go, 1 in nomad/fsm.go, 1 in nomad/alloc_endpoint.go, 1 in nomad/plan_apply.go, etc. It'd be worth taking another pass over these.

The ease of missing one makes me wonder if we want a NewEvaluationForJob constructor function that takes a Job, instead of hand-whittling the evaluation each time? That'd let us make sure all the appropriate fields are getting set.

@tgross
Copy link
Member

tgross commented Jul 14, 2025

Summary of our offline discussion: Having the node pool in the "root" of the eval isn't something we can use right now, and most consumers of it in the future will probably have the structs.Job handy anyway and can just grab the node pool from that. And that means we're probably just carting around for no good reason, and everything we persist here is going to get written to Raft and be hard to remove. Adding it to the FailedTGAllocs as in your original design will only add to state when we need it instead of for every allocation. And that matches up more nicely to how the existing code handles DC metrics. So we're going to revert back to your original design. (:blush: for my bad suggestions)

Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! :shipit:

@allisonlarson allisonlarson merged commit 3ca518e into main Jul 15, 2025
44 checks passed
@allisonlarson allisonlarson deleted the f-add-node-pool-to-metrics branch July 15, 2025 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/1.10.x backport to 1.10.x release line theme/docs Documentation issues and enhancements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: Support for node_pool Label in certain server metrics
4 participants