Add command to dump frontend dynamic config values #8214

vaibhavyadav-dev · 2025-08-19T20:15:25Z

What changed?

Added an admin command that dump all frontend dynamic config values,

Why?

Operators currently cannot easily inspect frontend runtime dynamic configuration, dumping values aids troubleshooting, incident response, and verification after config changes.

How did you test it?

Closes #421

dnr · 2025-08-19T20:58:43Z

This isn't dumping all dynamic config values, it's only dumping the frontend config, and only for one particular (nonexistent) namespace. To do this properly you'd need to add support in the dynamicconfig.Client interface, but I'm not sure we want to do that. We could support getting the value of a single key by name without changing the interface.

What are you actually trying to use this for?

bergundy · 2025-08-19T21:49:31Z

@dnr, @vaibhavyadav-dev joined the community slack and volunteered to contribute. I gave him this issue since I knew there's been a longstanding ask to add this API.

100% agree with the feedback so far. I can guide from here.

bergundy

You are going to want to also add a ListKeys() method to the dynamic config client.

@dnr do you think this is a reasonable request?

bergundy · 2025-08-19T21:50:15Z

proto/internal/temporal/server/api/adminservice/v1/service.proto


    rpc ForceUnloadTaskQueuePartition (ForceUnloadTaskQueuePartitionRequest) returns (ForceUnloadTaskQueuePartitionResponse) {}
+
+    rpc GetConfigurations(GetConfigurationsRequest) returns (GetConfigurationsResponse) {}


Suggested change

rpc GetConfigurations(GetConfigurationsRequest) returns (GetConfigurationsResponse) {}

rpc GetConfiguration(GetConfigurationRequest) returns (GetConfigurationResponse) {}

bergundy · 2025-08-19T21:50:29Z

proto/internal/temporal/server/api/adminservice/v1/request_response.proto

+message GetConfigurationsRequest {}
+message GetConfigurationsResponse{


Suggested change

message GetConfigurationsRequest {}

message GetConfigurationsResponse{

message GetConfigurationRequest {}

message GetConfigurationResponse {

Please make all this stuff have the string "DynamicConfig" in it. There are two types of configuration and we should be explicit to avoid confusion. (I would even prefer GetDynamicConfig to GetDynamicConfiguration since that's how it's referred to all over the code.)

bergundy · 2025-08-19T21:55:33Z

proto/internal/temporal/server/api/adminservice/v1/request_response.proto

+  string cluster_name = 1;
+  map<string, google.protobuf.Value> config_values = 2;


Suggested change

string cluster_name = 1;

map<string, google.protobuf.Value> config_values = 2;

repeated HostConfig host_config = 1;

And add a HostConfig message that will embed structured config via a struct:

message HostConfig { // The host that this configuration was loaded from. string hostname = 1; // The raw dynamic config loaded with all constraints. google.protobuf.Struct raw_dynamic_config = 2; }

dnr · 2025-08-19T22:41:32Z

You are going to want to also add a ListKeys() method to the dynamic config client.

@dnr do you think this is a reasonable request?

I would rather not. I don't think we should add methods there without a very good reason, and I don't feel like this is it. I think we should support a single key query to start and then see if it satisfies people's use case. (Tbh I'm skeptical this is even useful at all but I guess it doesn't hurt.)

The problem is that it'll break the build for anyone with a custom Client until they add support. Also just ListKeys plus calling GetValue on each key is racy (a key may be added or removed between the List and the Get). So you want to get a snapshot of the whole state, but that may be expensive.

If we really do need to do that, we should add it as optional (i.e. interface-assert it when needed) to not interfere with existing Clients.

bergundy · 2025-08-24T05:30:57Z

If we really do need to do that, we should add it as optional (i.e. interface-assert it when needed) to not interfere with existing Clients.

Curious if there are known client implementations that we would be concerned with here or if this is mostly speculation.
Agree that the interface assert would be cleaner.
100% can go with a Snapshot() method if needed. I think that would be a reasonable approach.
I'm thinking that this API would be good for debugging, I would want this to evolve to a point where we could potentially spot misaligned configurations between the nodes and it may come in handy for when we migrate between self-hosted and cloud.

bergundy · 2025-09-02T18:10:54Z

Lets make GetConfigurationRequest take a repeated string dynamic_config_keys and make the response include the repeated HostConfig as I've mentioned before but only for the current host, no need to fan out for now.

You will need to copy the definition of Constraints from common/dynamicconfig/client.go to proto/internal/temporal/server/api/config/v1/message.proto along with HostConfig and ConstrainedValue detailed below:

message ConstrainedValue {
  Constraints constraints = 1;
  google.protobuf.Struct value = 2;
}

message HostConfig {
  // The host that this configuration was loaded from.
  string hostname = 1;
  // The raw dynamic config loaded with all constraints.
  map<string, ConstrainedValue> dynamic_config = 2;
}

bergundy · 2025-09-17T15:24:58Z

Sorry if I mislead you, each config name may have multiple constrained values, this would be a better representation:

message ConstrainedValue {
  Constraints constraints = 1;
  google.protobuf.Struct value = 2;
}

message ConstrainedValues {
  repeated ConstrainedValue items = 1;
}

message HostConfig {
  // The host that this configuration was loaded from.
  string hostname = 1;
  // The raw dynamic config loaded with all constraints.
  map<string, ConstrainedValues> dynamic_config = 2;
}

bergundy · 2025-10-07T00:02:47Z

proto/internal/temporal/server/api/adminservice/v1/request_response.proto

+message Constraints {
+	string namespace = 1;
+	string namespace_id = 2;
+	string task_queue_name = 3;
+	int32 task_queue_type = 4;
+	int32 shard_id = 5;
+	int32 task_type = 6;
+	string destination = 7;
+}
+
+message ConstrainedValue {
+  Constraints constraints = 1;
+  google.protobuf.Struct value = 2;
+}
+
+message ConstrainedValues {
+  repeated ConstrainedValue items = 1;
+}


You can put this in proto/internal/temporal/server/api/dynamicconfig/v1/message.proto.

bergundy · 2025-10-07T00:05:50Z

proto/internal/temporal/server/api/adminservice/v1/service.proto


    rpc ForceUnloadTaskQueuePartition (ForceUnloadTaskQueuePartitionRequest) returns (ForceUnloadTaskQueuePartitionResponse) {}
+
+    rpc GetDynamicConfigurations(GetDynamicConfigurationsRequest) returns (GetDynamicConfigurationsResponse) {}


Suggested change

rpc GetDynamicConfigurations(GetDynamicConfigurationsRequest) returns (GetDynamicConfigurationsResponse) {}

rpc GetHostConfiguration(GetHostConfigurationRequest) returns (GetHostConfigurationResponse) {}

bergundy · 2025-10-07T00:06:38Z

service/frontend/admin_handler.go

+	pageSize := 100
+	var nextPageToken []byte
+
+	namespaces := make([]string, 0)


You don't need to get the list of namespaces to implement this feature.

bergundy · 2025-10-07T00:08:59Z

service/frontend/admin_handler.go

+		constrainedValues := adh.getConstrainedValueForKey(key, namespaces)
+		if len(constrainedValues) > 0 {
+			dynamicConfig[key] = &adminservice.ConstrainedValues{
+				Items: constrainedValues,
+			}
+		}


This would just be:

dcClient.GetValue(key)

Just make sure you have access to a dynamicconfig.Client instance in this handler.

vaibhavyadav-dev · 2025-11-21T10:20:17Z

Closing this PR; opening a fresh one with a cleaner commit history.
New PR

vaibhavyadav-dev requested a review from a team as a code owner August 19, 2025 20:15

bergundy reviewed Aug 19, 2025

View reviewed changes

vaibhavyadav-dev changed the title ~~add command to dump dynamic config values~~ added command to dump frontend dynamic config values Aug 22, 2025

bergundy mentioned this pull request Sep 9, 2025

[Feature Request] Service details command #8303

Closed

vaibhavyadav-dev force-pushed the added-tdbg-command-to-dump-dynamic-config-values branch from e90aa6c to adbce8b Compare September 22, 2025 19:29

bergundy changed the title ~~added command to dump frontend dynamic config values~~ Add command to dump frontend dynamic config values Sep 27, 2025

bergundy reviewed Oct 7, 2025

View reviewed changes

vaibhavyadav-dev requested a review from a team as a code owner November 14, 2025 18:00

vaibhavyadav-dev closed this Nov 21, 2025

vaibhavyadav-dev force-pushed the added-tdbg-command-to-dump-dynamic-config-values branch from 36962cc to 2cd597c Compare November 21, 2025 04:50


		rpc ForceUnloadTaskQueuePartition (ForceUnloadTaskQueuePartitionRequest) returns (ForceUnloadTaskQueuePartitionResponse) {}

		rpc GetConfigurations(GetConfigurationsRequest) returns (GetConfigurationsResponse) {}

	rpc GetConfigurations(GetConfigurationsRequest) returns (GetConfigurationsResponse) {}
	rpc GetConfiguration(GetConfigurationRequest) returns (GetConfigurationResponse) {}

		message GetConfigurationsRequest {}
		message GetConfigurationsResponse{

		string cluster_name = 1;
		map<string, google.protobuf.Value> config_values = 2;

	string cluster_name = 1;
	map<string, google.protobuf.Value> config_values = 2;
	repeated HostConfig host_config = 1;


		rpc ForceUnloadTaskQueuePartition (ForceUnloadTaskQueuePartitionRequest) returns (ForceUnloadTaskQueuePartitionResponse) {}

		rpc GetDynamicConfigurations(GetDynamicConfigurationsRequest) returns (GetDynamicConfigurationsResponse) {}

	rpc GetDynamicConfigurations(GetDynamicConfigurationsRequest) returns (GetDynamicConfigurationsResponse) {}
	rpc GetHostConfiguration(GetHostConfigurationRequest) returns (GetHostConfigurationResponse) {}

Add command to dump frontend dynamic config values #8214

Add command to dump frontend dynamic config values #8214

Uh oh!

Conversation

vaibhavyadav-dev commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Uh oh!

dnr commented Aug 19, 2025

Uh oh!

bergundy commented Aug 19, 2025

Uh oh!

bergundy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnr commented Aug 19, 2025

Uh oh!

bergundy commented Aug 24, 2025

Uh oh!

bergundy commented Sep 2, 2025

Uh oh!

bergundy commented Sep 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vaibhavyadav-dev commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vaibhavyadav-dev commented Aug 19, 2025 •

edited

Loading