[Bug] context deadline exceeded in integration tests

[jpds]

Describe the bug

I've created an integration test for Temporal for NixOS over at: https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/temporal.nix

This creates a fresh Temporal server in a VM, and then in the testScript section uses the cli to query that server about cluster status and workflows.

For some reason, the CLI calls will randomly fail with, whilst succeeding with a simple reattempt:

waiting for success: temporal workflow list --output json
... level=ERROR msg="failed reaching server: context deadline exceeded"

I've also tried setting --connect-timeout 120s, and it happens with that too.

This has also been encountered in NixOS's CI system: https://hydra.nixos.org/build/306058770/log

Minimal Reproduction

1. Install Nix
2. git clone https://github.com/NixOS/nixpkgs # Large download
3. cd nixpkgs; nix-build -A nixosTests.temporal

Environment/Versions

• OS and processor: Linux
• Temporal Version: [e.g. 1.14.0?] and/or SDK version: 1.28.1
• Are you using Docker or Kubernetes or building Temporal from source? Source

[Sushisource]

Hey @jpds, thanks for the report!

It might help to enable debug-level logging to see what's going on both server and cli side. You can pass --log-level debug for the cli, and it looks like you already have a spot in your flake to configure the server logging.

Also it looks like the CI is failing on namespace create rather than workflow list. Does it just randomly fail at some point on some command, or is it consistent? You may want to give server a bit more time to start up before calling the CLI operations (though, you are already doing some waiting, and in either case I would expect this to work) as a workaround.

Also, just for my curiosity, can you give us a little more background on why you're doing this? Would love to hear more.

[jpds]

It might help to enable debug-level logging to see what's going on both server and cli side. You can pass --log-level debug for the cli, and it looks like you already have a spot in your flake to configure the server logging.

Well, I just got this coredump:

temporal: (finished: waiting for success: temporal operator namespace create --namespace default --command-timeout 120s --log-level debug, in 0.13 seconds)
temporal: Namespace default successfully registered.

temporal: waiting for success: journalctl -o cat -u temporal.service | grep 'Register namespace succeeded'
temporal: (finished: waiting for success: journalctl -o cat -u temporal.service | grep 'Register namespace succeeded', in 0.07 seconds)
temporal: waiting for success: temporal operator namespace list --output json --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal operator namespace list --output json --command-timeout 120s --log-level debug, in 0.12 seconds)
temporal: waiting for success: temporal operator namespace describe --output json --namespace default --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal operator namespace describe --output json --namespace default --command-timeout 120s --log-level debug, in 0.12 seconds)
temporal: waiting for success: temporal workflow list --output json --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal workflow list --output json --command-timeout 120s --log-level debug, in 0.11 seconds)
temporal: waiting for success: temporal-hello-workflow.py
temporal # [   17.200608] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.492Z","msg":"Quota changed","service":"frontend","component":"long-poll-handler","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":1200,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   17.207538] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.494Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   17.211300] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.494Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   17.218914] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.497Z","msg":"Started Worker","service":"worker","Namespace":"default","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"temporal-system@temporal@default","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   17.225522] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.499Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:d98e32c0-b77a-4680-b9b2-55b999e44628","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   17.235397] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.501Z","msg":"Quota changed","service":"matching","component":"persistence","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   17.243180] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.505Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   17.248479] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.506Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   17.258391] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.508Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   17.264325] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.509Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   17.264800] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.511Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   17.265830] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.514Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   17.268980] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.520Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   17.270407] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:03.521Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   17.271287] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.522Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:d98e32c0-b77a-4680-b9b2-55b999e44628","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   17.272384] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.529Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   17.273820] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:03.530Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   17.888530] dhcpcd[786]: eth0: leased 10.0.2.15 for 86400 seconds
temporal # [   17.889387] dhcpcd[786]: eth0: adding route to 10.0.2.0/24
temporal # [   17.890464] dhcpcd[786]: eth0: adding default route via 10.0.2.2
temporal # [   18.079853] systemd[1]: Started DHCP Client.
temporal # [   18.082850] systemd[1]: Reached target Multi-User System.
temporal # [   18.085329] systemd[1]: Startup finished in 4.065s (kernel) + 14.018s (userspace) = 18.084s.
temporal # [   19.953257] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.243Z","msg":"Received StartWorkflowExecution.","service":"frontend","wf-id":"hello-activity-workflow-id","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:380"}
temporal # [   19.954880] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.243Z","msg":"Start workflow execution request namespace.","service":"frontend","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:384"}
temporal # [   19.957115] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.243Z","msg":"Start workflow execution request namespaceID.","service":"frontend","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:389"}
temporal # [   19.971438] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.263Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-scheduled-event-id":2,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"0001-01-01T00:00:00.000Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/service/history/workflow/workflow_task_state_machine.go:980"}
temporal # [   19.982148] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.269Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"VisibilityStartExecution","queue-task-id":1048590,"timestamp":"2025-08-28T23:36:06.269Z","cursor-timestamp":"2025-08-28T23:36:06.269Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.999742] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.269Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"TransferWorkflowTask","queue-task-id":1048591,"timestamp":"2025-08-28T23:36:06.269Z","cursor-timestamp":"2025-08-28T23:36:06.269Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.014332] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.269Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   20.020277] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.270Z","msg":"Quota changed","service":"history","component":"persistence","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   20.021405] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.271Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   20.022487] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.271Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.024186] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.271Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.025197] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.272Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.025812] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.272Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.043796] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.273Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.044890] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.273Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.045866] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.274Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   20.046994] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.275Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   20.047841] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.276Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.048751] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.276Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.049729] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.277Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.050842] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.277Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.051688] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.278Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   20.052728] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.281Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.053687] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.281Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.054696] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.282Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.055411] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.282Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"988@temporal-1b0cd3ab657f4d89b1e559ea15975926","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.055981] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.282Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.056940] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.283Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.057782] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.285Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","value":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639/hello-activity-workflow-id/0198f309-db76-74aa-8a05-32fbb56822a5","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   20.065391] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.289Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.150496] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.290Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   20.150694] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.294Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Task","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.151221] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.299Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   20.152501] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.353Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.153318] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.353Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.154233] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.353Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.154878] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.354Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.155733] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.354Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.156320] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.354Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.156947] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.354Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.158248] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.356Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   20.159364] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.363Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.161652] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.453Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.166659] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.458Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.175452] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.464Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.247281] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.466Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.249043] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.469Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.251228] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.543Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"988@temporal-1b0cd3ab657f4d89b1e559ea15975926","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.253197] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.545Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   20.287511] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.579Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":2,"wf-started-event-id":3,"workflow-task-request-id":"7da1ac47-cab4-4117-9918-f28883a9df99","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"2025-08-28T23:36:06.579Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   20.288609] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.580Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"WorkflowTaskTimeout","queue-task-id":1048595,"timestamp":"2025-08-28T23:36:16.580Z","cursor-timestamp":"2025-08-28T23:36:07.284Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.309464] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.601Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.310868] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.603Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.351772] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.644Z","msg":"RespondWorkflowTaskCompleted","service":"history","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":2,"logging-call-at":"go.temporal.io/server/service/history/handler.go:518"}
temporal # [   20.353227] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.645Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":0,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":0,"attempt":1,"wf-started-timestamp":"1970-01-01T00:00:00.000Z","wt-type":"Unspecified","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   20.356200] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.648Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"VisibilityUpsertExecution","queue-task-id":1048600,"timestamp":"2025-08-28T23:36:06.648Z","cursor-timestamp":"2025-08-28T23:36:07.284Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.357729] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.649Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"TransferActivityTask","queue-task-id":1048601,"timestamp":"2025-08-28T23:36:06.648Z","cursor-timestamp":"2025-08-28T23:36:07.284Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.365609] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.657Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","value":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639/hello-activity-workflow-id/0198f309-db76-74aa-8a05-32fbb56822a5","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   20.375167] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.666Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"ActivityTimeout","queue-task-id":1048602,"timestamp":"2025-08-28T23:36:16.667Z","cursor-timestamp":"2025-08-28T23:36:07.656Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.388153] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.678Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   20.404550] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.696Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":8,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"0001-01-01T00:00:00.000Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   20.413219] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.705Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"TransferWorkflowTask","queue-task-id":1048608,"timestamp":"2025-08-28T23:36:06.705Z","cursor-timestamp":"2025-08-28T23:36:07.656Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.413925] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.705Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"WorkflowTaskTimeout","queue-task-id":1048609,"timestamp":"2025-08-28T23:36:16.697Z","cursor-timestamp":"2025-08-28T23:36:07.656Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.432206] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.724Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","value":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639/hello-activity-workflow-id/0198f309-db76-74aa-8a05-32fbb56822a5","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   20.443120] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.734Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":8,"wf-started-event-id":9,"workflow-task-request-id":"1dd092c2-69c3-47e8-9e86-094c9d5c6f28","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"2025-08-28T23:36:06.734Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   20.443426] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.734Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"WorkflowTaskTimeout","queue-task-id":1048612,"timestamp":"2025-08-28T23:36:16.735Z","cursor-timestamp":"2025-08-28T23:36:07.723Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.451887] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.743Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   20.455387] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.745Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   20.472261] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.764Z","msg":"RespondWorkflowTaskCompleted","service":"history","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":8,"logging-call-at":"go.temporal.io/server/service/history/handler.go:518"}
temporal # [   20.477375] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.764Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","wf-scheduled-event-id":0,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":0,"attempt":1,"wf-started-timestamp":"1970-01-01T00:00:00.000Z","wt-type":"Unspecified","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   20.485958] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.765Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"TransferCloseExecution","queue-task-id":1048617,"timestamp":"2025-08-28T23:36:06.765Z","cursor-timestamp":"2025-08-28T23:36:07.723Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.495795] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.765Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"VisibilityCloseExecution","queue-task-id":1048618,"timestamp":"2025-08-28T23:36:06.765Z","cursor-timestamp":"2025-08-28T23:36:07.723Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.503807] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.765Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f309-db76-74aa-8a05-32fbb56822a5","queue-task-type":"DeleteHistoryEvent","queue-task-id":1048619,"timestamp":"2025-08-31T23:48:03.069Z","cursor-timestamp":"2025-08-28T23:36:07.723Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   20.511090] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.769Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639","value":"7ab87a9e-618f-49e8-bdaf-eac1ba1fa639/hello-activity-workflow-id/0198f309-db76-74aa-8a05-32fbb56822a5","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   20.534613] temporal-server[828]: {"level":"info","ts":"2025-08-28T23:36:06.826Z","msg":"Stopped physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"988@temporal-1b0cd3ab657f4d89b1e559ea15975926","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Stopped","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:265"}
temporal # [   20.542987] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:06.831Z","msg":"returning empty user data (expired)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":true,"request-known-version":0,"user-data-version":0,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:588"}
temporal # Fatal Python error: PyGILState_Release: thread state 0x7f887c025f10 must be current when releasing
temporal # Python runtime state: finalizing (tstate=0x00007f8887dd1dd0)
temporal # 
temporal # Thread 0x00007f888748a740 (most recent call first):
temporal #   <no Python frame>
temporal # [   20.640190] systemd-coredump[1041]: Process 988 (python3.13) of user 0 terminated abnormally with signal 6/ABRT, processing...
temporal # [   20.647926] systemd[1]: Created slice Slice /system/systemd-coredump.
temporal # [   20.653665] systemd[1]: Started Process Core Dump (PID 1041/UID 0).
temporal # [   21.179745] systemd-coredump[1042]: Process 988 (python3.13) of user 0 dumped core.
temporal # 
temporal # Module liblzma.so.5 without build-id.
temporal # Module libbz2.so.1 without build-id.
temporal # Module temporal_sdk_bridge.abi3.so without build-id.
temporal # Module _message.cpython-313-x86_64-linux-gnu.so without build-id.
temporal # Module libz.so.1 without build-id.
temporal # Module libgcc_s.so.1 without build-id.
temporal # Stack trace of thread 1035:
temporal # #0  0x00007f888752baac __pthread_kill_implementation (libc.so.6 + 0x9caac)
temporal # #1  0x00007f88874d090e raise (libc.so.6 + 0x4190e)
temporal # #2  0x00007f88874b7942 abort (libc.so.6 + 0x28942)
temporal # #3  0x00007f8887838a52 fatal_error.cold (libpython3.13.so.1.0 + 0x85a52)
temporal # #4  0x00007f88878b6ab8 _Py_FatalErrorFormat (libpython3.13.so.1.0 + 0x103ab8)
temporal # #5  0x00007f8887a926a6 PyGILState_Release (libpython3.13.so.1.0 + 0x2df6a6)
temporal # #6  0x00007f888542fb3f n/a (temporal_sdk_bridge.abi3.so + 0xe10b3f)
temporal # #7  0x00007f88847b8b59 n/a (temporal_sdk_bridge.abi3.so + 0x199b59)
temporal # #8  0x00007f888593287a n/a (temporal_sdk_bridge.abi3.so + 0x131387a)
temporal # #9  0x00007f88859406f6 n/a (temporal_sdk_bridge.abi3.so + 0x13216f6)
temporal # #10 0x00007f88859363c4 n/a (temporal_sdk_bridge.abi3.so + 0x13173c4)
temporal # #11 0x00007f888594da95 n/a (temporal_sdk_bridge.abi3.so + 0x132ea95)
temporal # #12 0x00007f8884b7f9f1 n/a (temporal_sdk_bridge.abi3.so + 0x5609f1)
temporal # #13 0x00007f888752997a start_thread (libc.so.6 + 0x9a97a)
temporal # #14 0x00007f88875b1d2c __clone3 (libc.so.6 + 0x122d2c)
temporal # 
temporal # Stack trace of thread 988:
temporal # #0  0x00007f88875af8bd syscall (libc.so.6 + 0x1208bd)
temporal # #1  0x00007f88848f926a n/a (temporal_sdk_bridge.abi3.so + 0x2da26a)
temporal # #2  0x00007f8885937720 n/a (temporal_sdk_bridge.abi3.so + 0x1318720)
temporal # #3  0x00007f888594c238 n/a (temporal_sdk_bridge.abi3.so + 0x132d238)
temporal # #4  0x00007f8884792222 n/a (temporal_sdk_bridge.abi3.so + 0x173222)
temporal # #5  0x00007f88847907c0 n/a (temporal_sdk_bridge.abi3.so + 0x1717c0)
temporal # #6  0x00007f888479bdc3 n/a (temporal_sdk_bridge.abi3.so + 0x17cdc3)
temporal # #7  0x00007f8884f3d5f8 n/a (temporal_sdk_bridge.abi3.so + 0x91e5f8)
temporal # #8  0x00007f8885364782 n/a (temporal_sdk_bridge.abi3.so + 0xd45782)
temporal # #9  0x00007f888496736c n/a (temporal_sdk_bridge.abi3.so + 0x34836c)
temporal # #10 0x00007f888540c2f3 n/a (temporal_sdk_bridge.abi3.so + 0xded2f3)
temporal # #11 0x00007f8887a985d4 PyObject_ClearManagedDict (libpython3.13.so.1.0 + 0x2e55d4)
temporal # #12 0x00007f8887a9aad0 subtype_dealloc (libpython3.13.so.1.0 + 0x2e7ad0)
temporal # #13 0x00007f8887a985d4 PyObject_ClearManagedDict (libpython3.13.so.1.0 + 0x2e55d4)
temporal # #14 0x00007f8887a9aad0 subtype_dealloc (libpython3.13.so.1.0 + 0x2e7ad0)
temporal # #15 0x00007f888786955a dictkeys_decref.part.0.constprop.0 (libpython3.13.so.1.0 + 0xb655a)
temporal # #16 0x00007f88878cb2df dict_tp_clear (libpython3.13.so.1.0 + 0x1182df)
temporal # #17 0x00007f8887aa1536 gc_collect_main (libpython3.13.so.1.0 + 0x2ee536)
temporal # #18 0x00007f8887b6cf51 _Py_Finalize.part.0.constprop.0 (libpython3.13.so.1.0 + 0x3b9f51)
temporal # #19 0x00007f8887b935a8 Py_RunMain (libpython3.13.so.1.0 + 0x3e05a8)
temporal # #20 0x00007f88874b94d8 __libc_start_call_main (libc.so.6 + 0x2a4d8)
temporal # #21 0x00007f88874b959b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a59b)
temporal # #22 0x0000000000401075 n/a (/iyff8129iampdw13nlfqalzhxy8y1hi9-python3-3.13.6/bin/python3.13 + 0x1075)
temporal # ELF object binary architecture: AMD x86-64
temporal # 
temporal # [   21.236513] systemd[1]: systemd-coredump@0-1041-0.service: Deactivated successfully.
temporal # [   21.242946] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:07.514Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6939,"hostId":"bf1238df-8467-11f0-8643-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   21.244556] systemd[1]: systemd-coredump@0-1041-0.service: Consumed 331ms CPU time, 68.7M memory peak, 5.7M written to disk.
temporal # timeout: the monitored command dumped core
temporal # [   21.840560] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:08.132Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6935,"hostId":"bf049722-8467-11f0-8643-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   24.200358] temporal-server[828]: {"level":"debug","ts":"2025-08-28T23:36:10.492Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6934,"hostId":"bef5e216-8467-11f0-8643-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}

Re-ran it, looks fairly boring otherwise:

temporal: waiting for success: journalctl -o cat -u temporal.service | grep 'Frontend is now healthy'
temporal: (finished: waiting for success: journalctl -o cat -u temporal.service | grep 'Frontend is now healthy', in 0.09 seconds)
temporal: waiting for success: temporal operator cluster list --output json --command-timeout 120s --log-level debug
temporal # [   13.212863] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.243Z","msg":"Quota changed","service":"frontend","component":"persistence","scope":"namespace","wf-namespace":"temporal-system","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   13.226673] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.257Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-tq-scanner-taskqueue-0","WorkerID":"temporal-system@127.0.0.1:7239","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   13.239934] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.270Z","msg":"Quota changed","service":"frontend","component":"long-poll-handler","scope":"namespace","wf-namespace":"temporal-system","current-quota":null,"new-quota":1200,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   13.246718] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.277Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.252518] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.281Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.259850] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.282Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.265440] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.282Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.271554] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.288Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.276730] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.288Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.283312] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.297Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.288822] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.297Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.297589] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.299Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.303204] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.300Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.308645] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.300Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.315682] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.301Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.320560] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.303Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.325652] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.305Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.331267] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.312Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.337492] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.314Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.343240] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.319Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.348667] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.320Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.354569] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.326Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.359110] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.326Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.365240] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.333Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.369488] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.344Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.374090] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.371Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.382337] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.400Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.386922] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.400Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.393119] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.408Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.400786] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.413Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:bca9fec3-7793-4b21-8a99-4ee8ef548581","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.408468] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.417Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.416657] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.423Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.424908] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.426Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.432490] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.442Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.436729] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.443Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.443192] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.447Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.452175] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.450Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.459726] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.456Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.468356] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.460Z","msg":"Quota changed","service":"matching","component":"persistence","scope":"namespace","wf-namespace":"temporal-system","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   13.474287] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.461Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.478445] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.462Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.484394] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.465Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.488264] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.466Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.494726] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.480Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   13.502237] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.486Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.507216] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.486Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.514283] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.500Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.518965] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.500Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.525127] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.507Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.529212] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.507Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.534987] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.516Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.539415] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.516Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.548172] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.525Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.552500] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.526Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.560296] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.538Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.569264] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.563Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   13.578090] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.574Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-history-scanner-taskqueue-0","WorkerID":"temporal-system@127.0.0.1:7239","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   13.584425] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.601Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:c7a05228-8c0b-4804-8a49-85300ad00b12","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.592988] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.607Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.602442] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.616Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.616666] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.625Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.627653] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.633Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.632048] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.653Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.638726] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.655Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.645632] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.658Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.653535] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.659Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.666546] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.660Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.678515] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.660Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.685622] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.662Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.694728] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.663Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.706344] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.663Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.715564] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.663Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.722712] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.664Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.733120] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.666Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.739562] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.666Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.750106] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.669Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   13.753501] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.670Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.759495] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.670Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.765565] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.671Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.770911] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.671Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.778865] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.676Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.785375] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.683Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-tq-scanner-taskqueue-0","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.791439] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.688Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.797694] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.694Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.804467] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.696Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:c7a05228-8c0b-4804-8a49-85300ad00b12","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.810296] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.698Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.816980] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.698Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.823439] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.702Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal: (finished: waiting for success: temporal operator cluster list --output json --command-timeout 120s --log-level debug, in 1.48 seconds)
temporal: [
{
  "clusterName": "active",
  "clusterId": "37c11d1f-57e2-4913-902c-3826e3671681",
  "address": "localhost:7233",
  "httpAddress": "localhost:7243",
  "initialFailoverVersion": "1",
  "historyShardCount": 1,
  "isConnectionEnabled": true
}
]

temporal: waiting for success: temporal operator cluster list --output json --command-timeout 120s --log-level debug
temporal # [   13.832341] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.703Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.837406] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.705Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.842935] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.705Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   13.850575] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.710Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.859431] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.712Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.868363] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.717Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.881112] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.720Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.893609] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.727Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.905300] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.729Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:bca9fec3-7793-4b21-8a99-4ee8ef548581","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   13.913222] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.751Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-processor-parent-close-policy","WorkerID":"temporal-system@127.0.0.1:7239","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   13.919713] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.753Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.925884] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.753Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.934652] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.758Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.941338] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.758Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.949900] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.759Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.955517] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.759Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.965806] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.761Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   13.972156] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.762Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   13.982160] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.764Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   13.995236] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.771Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:4ba045a1-2cec-4172-a4b9-38276648eed9","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.010561] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.772Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.023313] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.780Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.033450] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.788Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.042614] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.795Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.050484] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.798Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.059541] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.799Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.065784] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.803Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.070460] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.815Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   14.079240] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.818Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.087555] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.819Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.091612] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.819Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.098760] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.825Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.102844] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.826Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.109180] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.828Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.117774] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.828Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.123633] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.828Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.129328] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.832Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.134871] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.833Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:4ba045a1-2cec-4172-a4b9-38276648eed9","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.140408] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.831Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.146072] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.834Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-history-scanner-taskqueue-0","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.151311] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.835Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.157359] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.836Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.164488] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.840Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.171236] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.841Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.176909] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.848Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.182780] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.852Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   14.186066] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.852Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   14.194605] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.854Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.195278] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.855Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.195982] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.855Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.198486] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.865Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   14.199994] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.865Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   14.200847] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.879Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"default-worker-tq","WorkerID":"temporal-system@temporal","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   14.203210] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.880Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:cc1b2721-a968-4978-bcdd-bc6859978ba3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.204640] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.883Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.205877] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.888Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.207101] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.894Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.208140] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.911Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.208937] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.914Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.209798] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.926Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.210705] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.931Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.211648] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.936Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.217106] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.937Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-dlq-activity-tq","WorkerID":"temporal-system@temporal","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   14.220667] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.941Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.226002] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.943Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.232134] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.945Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.238620] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.949Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.240907] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.950Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   14.246786] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.954Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.252747] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.954Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.257740] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.953Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.263201] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.956Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.268675] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.956Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.273791] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.961Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.280446] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.962Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.286497] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.967Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.290220] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.967Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:cc1b2721-a968-4978-bcdd-bc6859978ba3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.295662] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.977Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.302090] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.982Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.308768] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.985Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.314095] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:56.992Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-migration-activity-tq","WorkerID":"temporal-system@temporal","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   14.318169] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.993Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.324403] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.995Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.328126] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:56.996Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.333365] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.000Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.336281] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.009Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-delete-namespace-activity-tq","WorkerID":"temporal-system@temporal","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   14.340271] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.010Z","msg":"none","component":"worker-manager","lifecycle":"Started","logging-call-at":"go.temporal.io/server/service/worker/worker.go:100"}
temporal # [   14.343216] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.011Z","msg":"none","component":"perns-worker-manager","lifecycle":"Starting","logging-call-at":"go.temporal.io/server/service/worker/pernamespaceworker.go:152"}
temporal # [   14.346709] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.011Z","msg":"none","component":"perns-worker-manager","lifecycle":"Started","logging-call-at":"go.temporal.io/server/service/worker/pernamespaceworker.go:164"}
temporal # [   14.349823] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.011Z","msg":"worker service started","service":"worker","component":"worker","address":"127.0.0.1:7239","logging-call-at":"go.temporal.io/server/service/worker/service.go:243"}
temporal # [   14.353758] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.011Z","msg":"OnStart hook executed","component":"fx","callee":"go.temporal.io/server/service/worker.(*Service).Start-fm()","caller":"go.temporal.io/server/service/worker.ServiceLifetimeHooks","runtime":"3.250131232s","logging-call-at":"go.temporal.io/server/temporal/fx.go:1033"}
temporal # [   14.358728] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.012Z","msg":"started","component":"fx","logging-call-at":"go.temporal.io/server/temporal/fx.go:1136"}
temporal # [   14.360814] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.012Z","msg":"OnStart hook executed","component":"fx","callee":"go.temporal.io/server/temporal.(*ServerImpl).Start-fm()","caller":"go.temporal.io/server/temporal.ServerLifetimeHooks","runtime":"3.475604581s","logging-call-at":"go.temporal.io/server/temporal/fx.go:1033"}
temporal # [   14.365930] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.012Z","msg":"started","component":"fx","logging-call-at":"go.temporal.io/server/temporal/fx.go:1136"}
temporal # [   14.368801] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.010Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.371985] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.016Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-migration-activity-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.377801] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.030Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.383902] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.034Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.389110] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.040Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.394375] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.043Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.401108] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.049Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.403980] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.050Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.410750] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.053Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.417279] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.064Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.422079] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.065Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-delete-namespace-activity-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.428654] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.068Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.433773] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.083Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   14.440670] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.084Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.445294] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.043Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.448991] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.102Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-migration-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.454833] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.103Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.459854] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.106Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.466143] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.109Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   14.469293] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.109Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   14.473894] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.113Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.479292] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.118Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-migration-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   14.484941] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.120Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   14.489562] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.120Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"temporal-system","wf-namespace-id":"32049b68-7872-4094-8e63-d0dd59896a83","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   14.493813] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.118Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.497369] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.123Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.502321] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.127Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-delete-namespace-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.508753] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.128Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.518869] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.130Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.521173] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.131Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.526313] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.131Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.532283] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.132Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-migration-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.537385] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.134Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-delete-namespace-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   14.591102] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.135Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-migration-activity-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.591980] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.137Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"temporal-system@temporal@temporal-system","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   14.592703] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.138Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:350fd377-ec20-4512-994d-49f2dfaecaa8","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.593522] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.142Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.594141] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.145Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.594720] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.145Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.595693] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.146Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"default-worker-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.596670] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.146Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/1","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.597610] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.146Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/2","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.598335] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.146Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-delete-namespace-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.598951] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.150Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.599735] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.155Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.600667] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.158Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.605284] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.168Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-dlq-activity-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.610856] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.178Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   14.616053] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.180Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.618942] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.180Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/default-worker-tq/3","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.623556] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.185Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-delete-namespace-activity-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.628697] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.187Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.634221] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.190Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/1","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.638925] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.192Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:350fd377-ec20-4512-994d-49f2dfaecaa8","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.644611] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:57.204Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   14.699606] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.195Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/2","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.700632] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.195Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-dlq-activity-tq/3","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   14.701240] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.196Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   14.701756] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.219Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"temporal-system","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   14.702546] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:57.222Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"temporal-system","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal: (finished: waiting for success: temporal operator cluster list --output json --command-timeout 120s --log-level debug, in 0.93 seconds)
temporal: waiting for success: temporal operator cluster describe --output json --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal operator cluster describe --output json --command-timeout 120s --log-level debug, in 0.12 seconds)
temporal: {
  "supportedClients": {
    "Nexus-go-sdk": "<2.0.0",
    "temporal-cli": "<2.0.0",
    "temporal-go": "<2.0.0",
    "temporal-java": "<2.0.0",
    "temporal-php": "<2.0.0",
    "temporal-server": "<2.0.0",
    "temporal-typescript": "<2.0.0",
    "temporal-ui": "<3.0.0"
  },
  "serverVersion": "1.28.1",
  "clusterId": "37c11d1f-57e2-4913-902c-3826e3671681",
  "clusterName": "active",
  "historyShardCount": 1,
  "persistenceStore": "sqlite",
  "visibilityStore": "sqlite"
}

temporal: waiting for success: temporal operator cluster describe --output json --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal operator cluster describe --output json --command-timeout 120s --log-level debug, in 0.10 seconds)
temporal: waiting for success: temporal operator namespace create --namespace default --command-timeout 120s --log-level debug
temporal # [   15.096358] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:58.127Z","msg":"Quota changed","service":"frontend","component":"rpc-handler","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":2400,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   15.096962] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:58.127Z","msg":"Quota changed","service":"frontend","component":"visibility-handler","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":10,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   15.097622] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:58.127Z","msg":"Quota changed","service":"frontend","component":"namespace-replication","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":10,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   15.107180] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:58.137Z","msg":"Quota changed","service":"frontend","component":"persistence","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   15.113777] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:58.144Z","msg":"Register namespace succeeded","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/namespace_handler.go:263"}
temporal: (finished: waiting for success: temporal operator namespace create --namespace default --command-timeout 120s --log-level debug, in 0.13 seconds)
temporal: Namespace default successfully registered.

temporal: waiting for success: journalctl -o cat -u temporal.service | grep 'Register namespace succeeded'
temporal: (finished: waiting for success: journalctl -o cat -u temporal.service | grep 'Register namespace succeeded', in 0.07 seconds)
temporal: waiting for success: temporal operator namespace list --output json --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal operator namespace list --output json --command-timeout 120s --log-level debug, in 0.11 seconds)
temporal: waiting for success: temporal operator namespace describe --output json --namespace default --command-timeout 120s --log-level debug
temporal: (finished: waiting for success: temporal operator namespace describe --output json --namespace default --command-timeout 120s --log-level debug, in 0.10 seconds)
temporal: waiting for success: temporal workflow list --output json --command-timeout 120s --log-level debug
temporal # [   15.453824] dhcpcd[784]: eth0: leased 10.0.2.15 for 86400 seconds
temporal # [   15.454716] dhcpcd[784]: eth0: adding route to 10.0.2.0/24
temporal # [   15.454999] dhcpcd[784]: eth0: adding default route via 10.0.2.2
temporal: (finished: waiting for success: temporal workflow list --output json --command-timeout 120s --log-level debug, in 0.14 seconds)
temporal: waiting for success: temporal-hello-workflow.py
temporal # [   15.631989] systemd[1]: Started DHCP Client.
temporal # [   15.635317] systemd[1]: Reached target Multi-User System.
temporal # [   15.636509] systemd[1]: Startup finished in 3.596s (kernel) + 12.038s (userspace) = 15.635s.
temporal # [   16.791229] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.821Z","msg":"Quota changed","service":"frontend","component":"long-poll-handler","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":1200,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   16.801558] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.824Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   16.806619] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.824Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   16.817332] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.827Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   16.821262] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.827Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   16.830871] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.827Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal:91aff8ca-5891-49fc-b331-5107d048843b","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   16.841477] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.828Z","msg":"Started Worker","service":"worker","Namespace":"default","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"temporal-system@temporal@default","logging-call-at":"sync/oncefunc.go:57"}
temporal # [   16.841771] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.829Z","msg":"Quota changed","service":"matching","component":"persistence","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   16.843198] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.829Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   16.844189] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.831Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   16.845344] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.833Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   16.846239] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.835Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   16.847227] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:39:59.854Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   16.848169] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.854Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   16.849096] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.854Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   16.849935] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.855Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal:91aff8ca-5891-49fc-b331-5107d048843b","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   16.851089] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:39:59.856Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"temporal-sys-per-ns-tq","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.220130] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.249Z","msg":"Received StartWorkflowExecution.","service":"frontend","wf-id":"hello-activity-workflow-id","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:380"}
temporal # [   19.230665] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.250Z","msg":"Start workflow execution request namespace.","service":"frontend","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:384"}
temporal # [   19.235747] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.250Z","msg":"Start workflow execution request namespaceID.","service":"frontend","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:389"}
temporal # [   19.242293] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.257Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-scheduled-event-id":2,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"0001-01-01T00:00:00.000Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/service/history/workflow/workflow_task_state_machine.go:980"}
temporal # [   19.248333] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.257Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"VisibilityStartExecution","queue-task-id":1048590,"timestamp":"2025-08-28T23:40:02.257Z","cursor-timestamp":"2025-08-28T23:40:02.257Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.254976] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.257Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"TransferWorkflowTask","queue-task-id":1048591,"timestamp":"2025-08-28T23:40:02.257Z","cursor-timestamp":"2025-08-28T23:40:02.257Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.275469] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.257Z","msg":"Quota changed","service":"history","component":"persistence","scope":"namespace","wf-namespace":"default","current-quota":null,"new-quota":0,"logging-call-at":"go.temporal.io/server/common/quotas/calculator/logged_calculator.go:100"}
temporal # [   19.280369] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.262Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   19.283922] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.265Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   19.287132] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.268Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   19.295409] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.270Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   19.299328] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.273Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   19.302633] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.277Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.305594] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.277Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.310101] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.279Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.313587] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.279Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.320166] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.280Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.324327] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.280Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.332732] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.281Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.336390] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.281Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.341903] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.282Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.348596] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.282Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.355085] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.284Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   19.364553] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.290Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"1025@temporal-5144632bbb244f7582c3a380fd4c849c","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.377517] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.294Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"UserData","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.379591] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.295Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.380984] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.295Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.387117] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.296Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.388950] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.296Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Activity","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Poll","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.391666] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.297Z","msg":"loaded user data from db","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","user-data-version":0,"timestamp":"1970-01-01T00:00:00.000Z","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:690"}
temporal # [   19.392084] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.301Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.392931] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.301Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.394189] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.301Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.395182] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.301Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.396136] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.301Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.396833] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.303Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.398117] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.304Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.451203] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.304Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.451803] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.304Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.452295] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.321Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.452706] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.305Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"1025@temporal-5144632bbb244f7582c3a380fd4c849c","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.453160] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.322Z","msg":"Started physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Started","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:244"}
temporal # [   19.453559] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.322Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.454096] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.323Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":2,"wf-started-event-id":3,"workflow-task-request-id":"23eb56dd-4cee-4a7b-87cb-c6215bace107","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"2025-08-28T23:40:02.323Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   19.454683] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.324Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"WorkflowTaskTimeout","queue-task-id":1048595,"timestamp":"2025-08-28T23:40:12.324Z","cursor-timestamp":"2025-08-28T23:40:03.286Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.455214] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.324Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/3","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.455617] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.325Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.456127] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.325Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Activity","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.457721] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.329Z","msg":"returning empty user data (no data)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":false,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:600"}
temporal # [   19.461803] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.330Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/2","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.466441] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.332Z","msg":"fetched user data from parent, no change","component":"matching-engine","wf-task-queue-name":"/_sys/hello-activity-task-queue/1","wf-task-queue-type":"Workflow","wf-namespace":"default","logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:312"}
temporal # [   19.470177] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.359Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.472861] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.359Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.533884] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.564Z","msg":"RespondWorkflowTaskCompleted","service":"history","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":2,"logging-call-at":"go.temporal.io/server/service/history/handler.go:518"}
temporal # [   19.534401] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.564Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":0,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":0,"attempt":1,"wf-started-timestamp":"1970-01-01T00:00:00.000Z","wt-type":"Unspecified","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   19.548495] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.579Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"VisibilityUpsertExecution","queue-task-id":1048600,"timestamp":"2025-08-28T23:40:02.579Z","cursor-timestamp":"2025-08-28T23:40:03.286Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.558456] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.579Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"TransferActivityTask","queue-task-id":1048601,"timestamp":"2025-08-28T23:40:02.579Z","cursor-timestamp":"2025-08-28T23:40:03.286Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.573310] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.603Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   19.583480] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.608Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"ActivityTimeout","queue-task-id":1048602,"timestamp":"2025-08-28T23:40:12.609Z","cursor-timestamp":"2025-08-28T23:40:03.600Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.591455] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.621Z","msg":"Received PollActivityTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:1073"}
temporal # [   19.614211] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.637Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":8,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"0001-01-01T00:00:00.000Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   19.622363] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.637Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"WorkflowTaskTimeout","queue-task-id":1048608,"timestamp":"2025-08-28T23:40:12.637Z","cursor-timestamp":"2025-08-28T23:40:03.600Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.632212] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.637Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"TransferWorkflowTask","queue-task-id":1048609,"timestamp":"2025-08-28T23:40:02.637Z","cursor-timestamp":"2025-08-28T23:40:03.600Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.639457] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.643Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   19.647487] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.647Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":8,"wf-started-event-id":9,"workflow-task-request-id":"a479a62d-a2e1-4670-a6a4-22c5d3ae794f","workflow-task-timeout":10,"attempt":1,"wf-started-timestamp":"2025-08-28T23:40:02.647Z","wt-type":"Normal","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   19.657396] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.648Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"WorkflowTaskTimeout","queue-task-id":1048612,"timestamp":"2025-08-28T23:40:12.648Z","cursor-timestamp":"2025-08-28T23:40:03.641Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.669383] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.663Z","msg":"Received PollWorkflowTaskQueue","service":"frontend","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:829"}
temporal # [   19.675237] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.663Z","msg":"Poll workflow task queue.","service":"frontend","wf-namespace":"default","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","logging-call-at":"go.temporal.io/server/service/frontend/workflow_handler.go:865"}
temporal # [   19.685458] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.672Z","msg":"RespondWorkflowTaskCompleted","service":"history","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":8,"logging-call-at":"go.temporal.io/server/service/history/handler.go:518"}
temporal # [   19.695540] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.673Z","msg":"Workflow task updated","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","wf-scheduled-event-id":0,"wf-started-event-id":0,"workflow-task-request-id":"emptyUuid","workflow-task-timeout":0,"attempt":1,"wf-started-timestamp":"1970-01-01T00:00:00.000Z","wt-type":"Unspecified","logging-call-at":"go.temporal.io/server/common/log/lazy_logger.go:29"}
temporal # [   19.706577] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.674Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"DeleteHistoryEvent","queue-task-id":1048617,"timestamp":"2025-08-31T23:57:00.814Z","cursor-timestamp":"2025-08-28T23:40:03.641Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.716144] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.674Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"TransferCloseExecution","queue-task-id":1048618,"timestamp":"2025-08-28T23:40:02.674Z","cursor-timestamp":"2025-08-28T23:40:03.641Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.727670] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.674Z","msg":"Assigning new task key","shard-id":1,"address":"127.0.0.1:7234","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","wf-id":"hello-activity-workflow-id","wf-run-id":"0198f30d-7550-7afd-adbc-0ecce3687149","queue-task-type":"VisibilityCloseExecution","queue-task-id":1048619,"timestamp":"2025-08-28T23:40:02.674Z","cursor-timestamp":"2025-08-28T23:40:03.641Z","logging-call-at":"go.temporal.io/server/service/history/shard/task_key_generator.go:98"}
temporal # [   19.739912] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.684Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"transfer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   19.750679] temporal-server[826]: {"level":"info","ts":"2025-08-28T23:40:02.754Z","msg":"Stopped physicalTaskQueueManager","component":"matching-engine","wf-task-queue-name":"1025@temporal-5144632bbb244f7582c3a380fd4c849c","wf-task-queue-type":"Workflow","wf-namespace":"default","worker-build-id":"_unversioned_","lifecycle":"Stopped","cause":"Force","logging-call-at":"go.temporal.io/server/service/matching/physical_task_queue_manager.go:265"}
temporal # [   19.759346] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:02.766Z","msg":"returning empty user data (expired)","component":"matching-engine","wf-task-queue-name":"hello-activity-task-queue","wf-task-queue-type":"Workflow","wf-namespace":"default","long-poll":true,"request-known-version":0,"user-data-version":0,"logging-call-at":"go.temporal.io/server/service/matching/user_data_manager.go:588"}
temporal: (finished: waiting for success: temporal-hello-workflow.py, in 4.33 seconds)
temporal: waiting for success: temporal workflow list --output json --command-timeout 120s --log-level debug
temporal # [   20.751705] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:03.781Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6934,"hostId":"4bc9a035-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   22.621522] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:05.651Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6935,"hostId":"4be4238b-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   22.769662] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:05.799Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6939,"hostId":"4c0253d8-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   22.800882] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:05.831Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6933,"hostId":"4bf0e3aa-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   29.298177] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:12.327Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"timer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   29.582129] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:12.612Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"timer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   29.608453] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:12.639Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"timer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # [   29.619330] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:12.649Z","msg":"Process task as active.","shard-id":1,"address":"127.0.0.1:7234","component":"timer-queue-processor","wf-namespace-id":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab","value":"6fa46cff-fd69-4d4a-9a54-dd1177e736ab/hello-activity-workflow-id/0198f30d-7550-7afd-adbc-0ecce3687149","logging-call-at":"go.temporal.io/server/service/history/queues/active_standby_executor.go:70"}
temporal # time=2025-08-28T23:40:13.016 level=ERROR msg="failed reaching server: context deadline exceeded"
temporal # [   32.771464] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:15.801Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6939,"hostId":"4c0253d8-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   33.622991] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:16.653Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6935,"hostId":"4be4238b-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   33.803383] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:16.832Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6933,"hostId":"4bf0e3aa-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   34.752988] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:17.783Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6934,"hostId":"4bc9a035-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # time=2025-08-28T23:40:24.112 level=ERROR msg="failed reaching server: context deadline exceeded"
temporal # [   44.805706] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:27.835Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6933,"hostId":"4bf0e3aa-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   46.624371] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:29.654Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6935,"hostId":"4be4238b-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   46.772433] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:29.802Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6939,"hostId":"4c0253d8-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # [   48.754815] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:31.784Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6934,"hostId":"4bc9a035-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}
temporal # time=2025-08-28T23:40:35.214 level=ERROR msg="failed reaching server: context deadline exceeded"
temporal # [   56.807310] temporal-server[826]: {"level":"debug","ts":"2025-08-28T23:40:39.837Z","msg":"Membership heartbeat upserted successfully","address":"127.0.0.1","port":6933,"hostId":"4bf0e3aa-8468-11f0-888a-525400123456","logging-call-at":"go.temporal.io/server/common/membership/ringpop/monitor.go:254"}

Also it looks like the CI is failing on namespace create rather than workflow list. Does it just randomly fail at some point on some command, or is it consistent?

It's so sporadic - I've seen it fail on most of them, but the workflow list right after the temporal-hello-workflow.py is usually the one that chokes up.

Also, just for my curiosity, can you give us a little more background on why you're doing this? Would love to hear more.

I don't know how familiar you are with NixOS, but: in addition to fully declarative systems, packages, and services - there's an entire integration testing framework built into the distribution. These tests are ran (potentitally in addition to unittests upstream developers provide) when packages are updated.

For instance, if we update AlertManager to a new release, does Prometheus still succeed in sending a JSON payload with an alert to it? When we capture that payload to a log - does it still match what we expect? This provides an additonal layer of assurance that an update isn't going to go on to break an end-user system. All of that is defined right here - and the entirity of it is automated, from the VM setup to the arbitary test commands.

If we update systemd-networkd - will we break people's routers' IPv6 setups at home with their ISP - we can just simulate that in a bunch of VMs.

And that's what I want end-to-end with all the Temporal components.

[Sushisource]

Cool to know about the tests, that's great!

The core dump is super unusual. Is this environment using musl by chance? Whatever that is, it's totally unrelated to the timeouts you were seeing before, as that's coming from the Python SDK and not the CLI.

Will this line https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/temporal.nix#L17 always pull in our most-recently-published SDK?

[jpds]

I did think the coredump was a fluke, glibc is the default (and there's a line about it in the stacktrace).

Will this line https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/temporal.nix#L17 always pull in our most-recently-published SDK?

That pulls in what's here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/temporalio/default.nix#L25

[Sushisource]

🤔 Hmm, I'm a bit at a loss then. I'd have to imagine there's something environment-sensitive here since we've got plenty of tests doing very similar stuff that don't have these problems.

I suspect you could probably work around this fairly easily by just keeping the timeout brief and retrying a few times. It's a bit of an unsatisfying answer but unfortunately I don't think we have the time to look into this otherwise since we've not got reports of this sort of thing elsewhere.

[jpds]

I suspect you could probably work around this fairly easily by just keeping the timeout brief and retrying a few times.

The tests are already set to retry.

I haven't made any changes to the test code, however they have been successful twice since I filed this: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.tests.temporal.x86_64-linux/all . Certainly, an odd one.

For now, I'm going to extend the test to include the Cassandra and see what happens.