Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 26 additions & 11 deletions docs/fern/pages/reference/components/tls-configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: TCP TLS
subtitle: Encrypt TCP streaming connections between frontend and workers
subtitle: Encrypt all TCP connections between frontend and workers
Comment thread
walkoss marked this conversation as resolved.
Resolved
---

Dynamo supports opt-in TLS encryption on the TCP call-home streaming transport
(implemented by `TcpStreamServer` and `TcpClient`, handling both response
streams and request streams between frontends and workers). When enabled, all TCP
connections on this path are upgraded to TLS using
[rustls](https://github.com/rustls/rustls) with the `ring` cryptographic
provider. When no TLS configuration is provided, the transport operates in
plaintext exactly as before.
Dynamo supports opt-in TLS encryption on all TCP transports between frontends
and workers. When enabled, both the **request plane** (frontend → worker
Comment thread
walkoss marked this conversation as resolved.
Resolved
inference requests) and the **response stream** (worker → frontend inference
output) are encrypted using [rustls](https://github.com/rustls/rustls) with the
`ring` cryptographic provider. When no TLS configuration is provided, all
transports operate in plaintext exactly as before.

The same `DYN_TCP_TLS_*` environment variables apply to both transports — a
single set of certificates encrypts all TCP traffic between Dynamo components.

## Environment variables

Expand Down Expand Up @@ -135,12 +137,25 @@ server and client depending on the stream direction.
> allowing TLS to be configured once at the platform level and auto-injected
> into all DGD pods without per-component env var setup.

## Encrypted paths

When TLS is configured, all TCP communication between frontend and worker is
encrypted:

| Path | Direction | Data | Transport |
|---|---|---|---|
| Request plane | Frontend → Worker | User prompts, request metadata | `egress/tcp_client` → `ingress/shared_tcp_endpoint` |
| Response stream | Worker → Frontend | Inference output tokens | `tcp/client` → `tcp/server` |
| Request stream | Frontend → Worker | Streaming input (bidirectional) | `tcp/client` → `tcp/server` |

## Design notes

- TLS configuration is cached after the first TCP connection via `OnceCell`.
- TLS configuration is cached after the first connection via `OnceCell`.
Certificate rotation requires a process restart.
- The TLS handshake is spawned per-connection on the server side so the accept
loop is never blocked by a slow handshake.
- The TLS handshake is spawned per-connection on both the request plane and
response stream servers so the accept loop is never blocked.
- Invalid TLS configuration on the request plane (e.g. bad cert path) prevents
server startup rather than silently falling back to plaintext.
Comment thread
walkoss marked this conversation as resolved.
Resolved
- When server and client TLS configurations are mismatched (e.g., server has TLS
but client does not), a warning is logged at startup.
- An empty CA certificate file is detected at load time and rejected with a
Expand Down
54 changes: 50 additions & 4 deletions lib/runtime/src/pipeline/network/egress/tcp_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,12 @@ use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::Duration;
use tokio::io::{AsyncWrite, AsyncWriteExt};
use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio_rustls::TlsConnector;

type BoxRead = Box<dyn AsyncRead + Unpin + Send>;
type BoxWrite = Box<dyn AsyncWrite + Unpin + Send>;
use tokio::sync::oneshot;
use tokio::task::JoinHandle;
use tokio_util::codec::FramedRead;
Expand Down Expand Up @@ -376,6 +380,27 @@ struct TcpConnection {
post_enqueue_barrier: Option<Arc<tokio::sync::Barrier>>,
}

/// Cached TLS connector for the request plane. Built once from env vars.
static REQUEST_PLANE_TLS_CONNECTOR: once_cell::sync::OnceCell<Option<TlsConnector>> =
once_cell::sync::OnceCell::new();

fn get_request_plane_tls_connector() -> anyhow::Result<&'static Option<TlsConnector>> {
REQUEST_PLANE_TLS_CONNECTOR.get_or_try_init(|| {
use crate::config::environment_names::tcp_response_stream::tls as env;
let ca_cert_path = std::env::var(env::DYN_TCP_TLS_CA_CERT_PATH).ok();
let insecure = crate::config::env_is_truthy(env::DYN_TCP_TLS_INSECURE);
let tls_requested = ca_cert_path.is_some() || insecure;
if !tls_requested {
return Ok(None);
}
Comment thread
walkoss marked this conversation as resolved.
Resolved
let tls_config = crate::tls_utils::client_tls_config(
ca_cert_path.as_deref().map(std::path::Path::new),
insecure,
)?;
Ok(Some(TlsConnector::from(std::sync::Arc::new(tls_config))))
})
}

impl TcpConnection {
/// Create a new connection with lock-free submit and batched write/read tasks
async fn connect(addr: SocketAddr, timeout: Duration, channel_buffer: usize) -> Result<Self> {
Expand All @@ -386,7 +411,28 @@ impl TcpConnection {
// Configure socket for lower latency
Self::configure_socket(&stream)?;

let (read_half, write_half) = tokio::io::split(stream);
let (read_half, write_half): (BoxRead, BoxWrite) = if let Some(connector) =
get_request_plane_tls_connector()?
{
use crate::config::environment_names::tcp_response_stream::tls as env;
let server_name = match std::env::var(env::DYN_TCP_TLS_SERVER_NAME) {
Ok(name) => rustls::pki_types::ServerName::try_from(name)
.map_err(|e| anyhow::anyhow!("invalid TLS server name: {e}"))?,
Err(_) => rustls::pki_types::ServerName::IpAddress(addr.ip().into()),
};
let tls_stream = tokio::time::timeout(
crate::tls_utils::handshake_timeout(),
connector.connect(server_name, stream),
)
.await
.map_err(|_| anyhow::anyhow!("Request plane TLS handshake timed out to {}", addr))?
.map_err(|e| anyhow::anyhow!("Request plane TLS handshake failed to {}: {e}", addr))?;
let (r, w) = tokio::io::split(tls_stream);
(Box::new(r), Box::new(w))
} else {
let (r, w) = tokio::io::split(stream);
(Box::new(r), Box::new(w))
};

let submit_queue = Arc::new(SegQueue::new());
let response_queue = Arc::new(SegQueue::new());
Expand Down Expand Up @@ -557,7 +603,7 @@ impl TcpConnection {
/// - If a response races back before waiters are queued, the reader's
/// existing spin-wait covers that small handoff window
async fn writer_task(
mut write_half: tokio::io::WriteHalf<TcpStream>,
mut write_half: BoxWrite,
Comment thread
walkoss marked this conversation as resolved.
Resolved
submit_queue: Arc<SegQueue<PendingRequest>>,
response_queue: Arc<SegQueue<oneshot::Sender<Result<Bytes>>>>,
notify: Arc<tokio::sync::Notify>,
Expand Down Expand Up @@ -698,7 +744,7 @@ impl TcpConnection {
/// On exit (clean close or error), sets `healthy=false` and wakes the writer
/// via `writer_notify` so it can detect reader death and drain pending callers.
async fn reader_task(
read_half: tokio::io::ReadHalf<TcpStream>,
read_half: BoxRead,
response_queue: Arc<SegQueue<oneshot::Sender<Result<Bytes>>>>,
healthy: Arc<AtomicBool>,
writer_notify: Arc<tokio::sync::Notify>,
Expand Down
Loading
Loading