identity: The initial implementation code for node identity.#26291
Merged
identity: The initial implementation code for node identity.#26291
Conversation
…ion to support node identities. (#26052) When Nomad generates an identity for a node, the root key used to sign the JWT will be stored as a field on the node object and written to state. To provide fast lookup of nodes by their signing key, the node table schema has been modified to include the keyID as an index. In order to ensure a root key is not deleted while identities are still actively signed by it, the Nomad state has an in-use check. This check has been extended to cover node identities. Nomad node identities will have an expiration. The expiration will be defined by a TTL configured within the node pool specification as a time duration. When not supplied by the operator, a default value of 24hr is applied. On cluster upgrades, a Nomad server will restore from snapshot and/or replay logs. The FSM has therefore been modified to ensure restored node pool objects include the default value. The builtin "all" and "default" pools have also been updated to include this default value. Nomad node identities will be a new identity concept in Nomad and will exist alongside workload identities. This change introduces a new envelope identity claim which contains generic public claims as well as either a node or workload identity claims. This allows us to use a single encryption and decryption path, no matter what the underlying identity. Where possible node and workload identities will use common functions for identity claim generation. The new node identity has the following claims: * "nomad_node_id" - the node ID which is typically generated on the first boot of the Nomad client as a UUID within the "ensureNodeID" function. * "nomad_node_pool" - the node pool is a client configuration parameter which provides logical grouping of Nomad clients. * "nomad_node_class" - the node class is a client configuration parameter which provides scheduling constraints for Nomad clients. * "nomad_node_datacenter" - the node datacenter is a client configuration parameter which provides scheduling constraints for Nomad clients and a logical grouping method.
When a node heartbeats, the RPC handler will optionally generate an identity to return to the caller. The identity key ID will be stored in the node object, so we have tracking of keys in use. The state store has been updated to handle node status update requests that include a signing key ID. Rather than add another parameter into the function signature, the FSM function now takes the entire request object.
The authenticator process which performs RPC authentication has been modified to support node identities. Node identities are verified by ensuring the node ID as claimed has a node written to Nomad state. The client only and generic authenticate methods now support both node secret IDs and node identities. It uses uuid checking to attempt to parse either option. A new method has also been added to handle the specific RPCs that will optionally generate node identities. While a new authenticator method is not ideal, it is better than the alternative option for these RPCs to perform complex additional RPC context work in order to understand whether an identity should be generated. The TLS verification functionality has been pulled into its own method to avoid further code duplication.
…6165) When a Nomad client register or re-registers, the RPC handler will generate and return a node identity if required. When an identity is generated, the signing key ID will be stored within the node object, to ensure a root key is not deleted until it is not used. During normal client operation it will periodically heartbeat to the Nomad servers to indicate aliveness. The RPC handler that is used for this action has also been updated to conditionally perform identity generation. Performing it here means no extra RPC handlers are required and we inherit the jitter in identity generation from the heartbeat mechanism. The identity generation check methods are performed from the RPC request arguments, so they a scoped to the required behaviour and can handle the nuance of each RPC. Failure to generate an identity is considered terminal to the RPC call. The client will include behaviour to retry this error which is always caused by the encrypter not being ready unless the servers keyring has been corrupted.
…26184) The Nomad client will persist its own identity within its state store for restart persistence. The added benefit of using it over the filesystem is that it supports transactions. This is useful when considering the identity will be renewed periodically.
Nomad servers, if upgraded, can return node identities as part of the register and update/heartbeat response objects. The Nomad client will now handle this and store it as appropriate within its memory and statedb. The client will now use any stored identity for RPC authentication with a fallback to the secretID. This supports upgrades paths where the Nomad clients are updated before the Nomad servers.
) The Nomad client will have its identity renewed according to the TTL which defaults to 24h. In certain situations such as root keyring rotation, operators may want to force clients to renew their identities before the TTL threshold is met. This change introduces a client HTTP and RPC endpoint which will instruct the node to request a new identity at its next heartbeat. This can be used via the API or a new command. While this is a manual intervention step on top of the any keyring rotation, it dramatically reduces the initial feature complexity as it provides an asynchronous and efficient method of renewal that utilises existing functionality.
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR comprises the initial node identity code. It has been previously reviewed in stages on a PR by PR basis. Merging this code into main now allows us to get it into our nightly testing loop and makes it easier for engineers to test it when they want to.
This initial PR does not include changelog or documentation updates which will come at a later date.
Links
internal jira: https://hashicorp.atlassian.net/issues/NMD-763
internal design doc: https://docs.google.com/document/d/1MYjlFlOAmGHmWGC3VsrIMUL_VSgKwjLAq6GUymDY38M/edit?tab=t.0
Contributor Checklist
changelog entry using the
make clcommand.ensure regressions will be caught.
and job configuration, please update the Nomad website documentation to reflect this. Refer to
the website README for docs guidelines. Please also consider whether the
change requires notes within the upgrade guide.
Reviewer Checklist
backporting document.
in the majority of situations. The main exceptions are long-lived feature branches or merges where
history should be preserved.
within the public repository.