Temporal cli gives context deadline exceeded. Tctl does not

[esn89]

What are you really trying to do?

Use temporal to connect to the frontend via gRPC.

Describe the bug

Using temporal to talk to the frontend gives context deadline exceeded. Using tctl is fine.

Here it is working with tctl:

Here it is NOT working with temporal:

It is going to the SAME backend FQDN. I am showing env to illustrate that I do not have any TEMPORAL_ env vars set.

Minimal Reproduction

Install temporal like how I did on Kubernetes of 1.33.3-gke.1136000

helm install temporal temporalio/temporal --version 0.65.0

Install istio for ingress. DO NOT add the sidecar injection into the namespace. We do not want the mesh feature. Ingress only.

./istioctl install --set profile=minimal -y

Set up ingress gateway:

helm upgrade --install istio-ingressgateway --namespace istio-system istio-official/gateway --version 1.27.0 --values ingress-gateway.yaml

My ingress-gateway.yaml looks like this:

name: istio-ingressgateway
service:
  loadBalancerIP: $MY_INTERNAL_IP
  annotations:
    cloud.google.com/load-balancer-type: Internal
    networking.gke.io/internal-load-balancer-subnet: internal1
  ports:
  - name: status-port
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: temporal
    port: 7233
    protocol: TCP
    targetPort: 7233
autoscaling:
  maxReplicas: 10
  targetCPUUtilizationPercentage: 75
resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 1Gi
    cpu: "1"

Once it is up (you will need to have your TLS certs) for Gateway and Virtualservice:

gateways:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: web-temporal-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - web-temporal.mycompany.com
    port:
      name: temporal-mutual
      number: 443
      protocol: HTTPS
    tls:
      credentialName: istio-ingressgateway-tls
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: backend-temporal-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - backend-temporal.mycompany.com
    port:
      name: backend-temporal-mutual
      number: 7233
      protocol: HTTPS
    tls:
      credentialName: istio-ingressgateway-tls
      mode: MUTUAL

Environment/Versions

• OS and processor: M1 Mac 15.6.1
• Temporal Version: 1.28.1
• Using Kubernetes

Additional context

tctl version 1.18.0
temporal version 1.4.1 (Server 1.28.0, UI 2.39.0)

[cretz]

Thanks, we will triage for investigation.

In order to have a full standalone replication, can you clarify the steps you took to setup mTLS? Specifically, please add the steps to setup and produce test.key and test.cert from your screenshot and how to configure Istio with them.

I did not test, but I wonder if this is an SNI or cert validation issue. Looking at code, tctl by default it appears sets the SNI override for the TLS config as the host provided whereas temporal does not assume you want to override SNI. So with temporal, I wonder if passing in --tls-server-name backend-temporal.whatever or --tls-disable-host-verification help.

[esn89]

The mTLS setup I don't have for you, as my company provides everything.
We simply have a tool which allows us to request for a server cert and a client cert for that particular server.

Meanwhile, I will try adding those flags to the temporal cli and see if that makes it go.

Nope, despite those 2 flags, still getting the error.

[cretz]

I am afraid we need a full end-to-end replication with the exact steps including the cert generation. It might be the cert configuration that is causing the issue. We are not aware of any mTLS issues connecting to servers if the certs (and SNI override and such) are accurate. Ideally the steps do not include Istio if it's not a problem specific to that one product.

[nhoffman5]

I am unfortunately running into the same issue as @esn89 where I am able to hit my temporal-frontend service via grpcurl and tctl, but am getting a context deadline exceeded on the first couple of "cold" attempts followed by successes on "warmed" attempts while using temporal CLI.

Environment

• Temporal Server: 1.28.0 (installed via Helm Chart)
• Ingress: Istio Gateway → AWS NLB (TLS via Let’s Encrypt)
• Frontend ports: 443 (HTTPS protocol) and 7233 (gRPC protocol) both route to service port 7233
• tctl version: 1.18.0
• Temporal CLI version: 1.4.1

Observed Behavior

Temporal CLI

"Cold start" conditions, after around 1 minute of total inactivity (no workers polling, no UI access):

temporal operator cluster health --address <host>:443 --tls --tls-server-name <host> --namespace default

Fails ~5-10s after start with time=... level=ERROR msg="failed reaching server: context deadline exceeded"

Immediately retry once or twice, then it starts succeeding reliably (path warmed)

Same result if I skip TLS and point directly at :7233 (without --tls), so TLS termination isn’t the differentiator.

tctl

tctl --address <host>:443 --tls_server_name <host> --namespace default cluster health

Succeeds instantly:

temporal.api.workflowservice.v1.WorkflowService: SERVING

grpcurl

grpcurl -d '{}' <host>:443 grpc.health.v1.Health/Check succeeds instantly: { "status": "SERVING" }

K8s Port‑forward

kubectl port-forward svc/temporal-frontend -n temporal 7233

temporal ... --address localhost:7233 succeeds immediately.

Network Checks

• DNS (dig) ~10ms
• TCP+TLS handshake (openssl s_client -alpn h2) ~180ms