scheduler: skip setting preferred node when nodepool changes

[mismithhisler]

Description

Preferred node is used when a task group has an ephemeral disk, so we ideally stay on the same node. However if the jobs node pool changes, we should not select the current node as the preferred node, and let the scheduler decide which node to pick from the correct node pool.

Testing & Reproduction steps

Links

Fixes GH #26600

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[jrasell]

The change looks good to me, but seeing as @pkazmierczak has been deep in this code, I'd like for him to take a quick look.

It would be nice if we could tighten the use of must for the added test cases; such as:

		if err := h.Process(NewServiceScheduler, eval); err != nil {
			t.Fatalf("err: %v", err)
		}

		must.NoError(t, h.Process(NewServiceScheduler, eval))

PR also needs a changelog entry.

[pkazmierczak]

Great work, @mismithhisler, LGTM. As noted by James, a small refactoring of the tests would be nice and a changelog entry.

[tgross]

LGTM!

[tgross]

I'm now realizing this bug probably happens with datacenters too (if they're not overlapping sets between versions), because we use both pool and DC to get the set of eligible nodes.