Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
# August 29, 2025: ~9 minutes
- cluster-name: 'cluster-four'
go-test-args: '-timeout=25m'
go-test-run-regex: '^TestKgateway$$/^ExtProc$$|^TestKgateway$$/^ExtAuth$$|^TestKgateway$$/^PolicySelector$$|^TestKgateway$$/^Backends$$|^TestKgateway$$/^BackendTLSPolicies$$|^TestKgateway$$/^CSRF$$|^TestKgateway$$/^AutoHostRewrite$$|^TestKgateway$$/^LeaderElection$$|^TestKgateway$$/^ListenerPolicyProxyProtocol$$|^TestKgateway$$/^Compression$$'
go-test-run-regex: '^TestKgateway$$/^ExtProc$$|^TestKgateway$$/^ExtAuth$$|^TestKgateway$$/^PolicySelector$$|^TestKgateway$$/^Backends$$|^TestKgateway$$/^BackendTLSPolicies$$|^TestKgateway$$/^CSRF$$|^TestKgateway$$/^AutoHostRewrite$$|^TestKgateway$$/^LeaderElection$$|^TestKgateway$$/^ListenerPolicyProxyProtocol$$|^TestKgateway$$/^Compression$$|^TestParallelControllers$$'
localstack: 'false'
# August 29, 2025: ~10 minutes
- cluster-name: 'cluster-five'
Expand Down
44 changes: 44 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,50 @@
## Project Overview
kgateway is a **dual control plane** implementing the Kubernetes Gateway API for both Envoy and agentgateway. It's built on KRT (Kubernetes Declarative Controller Runtime from Istio) and uses a plugin-based architecture for extensibility.

## Dual Controller Architecture

### Controller Names & Isolation
kgateway supports **two independent controllers** that can run side-by-side:
- **Envoy Controller**: `kgateway.dev/kgateway` (defined in `wellknown.DefaultGatewayControllerName`)
- **Agentgateway Controller**: `kgateway.dev/agentgateway` (defined in `wellknown.DefaultAgwControllerName`)

**Critical Requirements:**
1. Controllers MUST always respect `GatewayClass.spec.controllerName` Classname can matter, in the case of waypoints, but its always more specific information
2. Controllers MUST NOT process resources belonging to the other controller
3. Enable flags (`EnableEnvoy`, `EnableAgentgateway`) MUST be honored at all layers

### How Controllers Are Isolated

**Translation/KRT Collections:**
- Gateway collections filter by controllerName at creation time
- Routes inherit filtering from their parent Gateways
- Policy attachment respects Gateway's controllerName

**XDS Generation:**
- `ProxySyncer` (Envoy): Only translates Gateways with envoy controllerName (filtered by `GatewaysForEnvoyTransformationFunc`)
- `AgwSyncer` (Agentgateway): Only translates Gateways with agw controllerName (filtered in `GatewayCollection`)

**Status Writing:**
- Status syncers write status entries namespaced by controllerName
- Route status has per-controller parent entries (multiple controllers can write status)
- Gateway status is owned by the single controlling controller

**Deployment:**
- Gateway reconciler checks enable flags before calling deployer
- Deployer selects chart based on Gateway's controllerName from GatewayClass
- Chart selection: envoy chart for `kgateway.dev/kgateway`, agentgateway chart for `kgateway.dev/agentgateway`

**Enable Flags:**
- `EnableEnvoy` (default: true): Controls if envoy ProxySyncer, StatusSyncer, and GatewayClass creation run
- `EnableAgentgateway` (default: true): Controls if agentgateway AgwSyncer, StatusSyncer, and GatewayClass creation run
- Gateway reconciler checks flags before deploying resources for each controller

### Key Files for Controller Filtering
- `pkg/krtcollections/policy.go:473`: Envoy Gateway collection filtering
- `pkg/agentgateway/translator/gateway_collection.go:218`: Agentgateway Gateway collection filtering
- `internal/kgateway/controller/gw_controller.go:272-293`: Gateway reconciler enable flag checks
- `internal/kgateway/deployer/gateway_parameters.go:376-378`: Chart selection based on controllerName

## Architecture (Read This First!)

### Translation Pipeline (3 phases)
Expand Down
19 changes: 5 additions & 14 deletions api/v1alpha1/kgateway/gateway_parameters_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ type KubernetesProxyConfig struct {
Deployment *ProxyDeployment `json:"deployment,omitempty"`

// Configuration for the container running Envoy.
// If agentgateway is enabled, the EnvoyContainer values will be ignored.
// If the Gateway uses a GatewayClass with controllerName: kgateway.dev/agentgateway,
// the EnvoyContainer values will be ignored.
//
// +optional
EnvoyContainer *EnvoyContainer `json:"envoyContainer,omitempty"`
Expand Down Expand Up @@ -676,13 +677,10 @@ func (in *StatsMatcher) GetExclusionList() []shared.StringMatcher {
return in.ExclusionList
}

// Agentgateway configures the agentgateway dataplane integration to be enabled if the `agentgateway` GatewayClass is used.
// Agentgateway configures the agentgateway dataplane integration.
// The agentgateway dataplane is automatically used when the Gateway references a GatewayClass
// with controllerName: kgateway.dev/agentgateway.
type Agentgateway struct {
// Whether to enable the extension.
//
// +optional
Enabled *bool `json:"enabled,omitempty"`

// Log level for the agentgateway. Defaults to info.
// Levels include "trace", "debug", "info", "error", "warn". See: https://docs.rs/tracing/latest/tracing/struct.Level.html
//
Expand Down Expand Up @@ -739,13 +737,6 @@ type Agentgateway struct {
ExtraVolumeMounts []corev1.VolumeMount `json:"extraVolumeMounts,omitempty"`
}

func (in *Agentgateway) GetEnabled() *bool {
if in == nil {
return nil
}
return in.Enabled
}

func (in *Agentgateway) GetLogLevel() *string {
if in == nil {
return nil
Expand Down
5 changes: 0 additions & 5 deletions api/v1alpha1/kgateway/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,6 @@ spec:
When set, the agent gateway will use this configmap instead of creating the default one.
The configmap must contain a 'config.yaml' key with the agent gateway configuration.
type: string
enabled:
description: Whether to enable the extension.
type: boolean
env:
description: The container environment variables.
items:
Expand Down Expand Up @@ -657,7 +654,8 @@ spec:
envoyContainer:
description: |-
Configuration for the container running Envoy.
If agentgateway is enabled, the EnvoyContainer values will be ignored.
If the Gateway uses a GatewayClass with controllerName: kgateway.dev/agentgateway,
the EnvoyContainer values will be ignored.
properties:
bootstrap:
description: Initial envoy configuration.
Expand Down
85 changes: 48 additions & 37 deletions pkg/deployer/deployer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,6 @@ var _ = Describe("Deployer", func() {
Spec: kgateway.GatewayParametersSpec{
Kube: &kgateway.KubernetesProxyConfig{
Agentgateway: &kgateway.Agentgateway{
Enabled: ptr.To(true),
Image: &kgateway.Image{
Tag: ptr.To("0.4.0"),
},
Expand Down Expand Up @@ -430,9 +429,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -516,9 +516,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err = deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -595,9 +596,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -701,9 +703,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -768,9 +771,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -867,9 +871,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -939,9 +944,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -1027,9 +1033,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -1121,9 +1128,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -1211,9 +1219,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d1, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand All @@ -1237,9 +1246,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: "bar",
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
})
d2, err := deployerinternal.NewGatewayDeployer(
wellknown.DefaultGatewayControllerName,
Expand Down Expand Up @@ -1910,9 +1920,10 @@ var _ = Describe("Deployer", func() {
Registry: "foo",
Tag: defaultImageTag,
},
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
GatewayClassName: wellknown.DefaultGatewayClassName,
WaypointGatewayClassName: wellknown.DefaultWaypointClassName,
AgentgatewayClassName: wellknown.DefaultAgwClassName,
AgentgatewayControllerName: wellknown.DefaultAgwControllerName,
}
}

Expand Down
50 changes: 29 additions & 21 deletions pkg/deployer/gateway_parameters.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,33 @@ func GetInMemoryGatewayParameters(cfg InMemoryGatewayParametersConfig) *kgateway
// set for the agentgateway deployment.
func defaultAgentgatewayParameters(imageInfo *ImageInfo, omitDefaultSecurityContext bool) *kgateway.GatewayParameters {
gwp := defaultGatewayParameters(imageInfo, omitDefaultSecurityContext)
gwp.Spec.Kube.Agentgateway.Enabled = ptr.To(true)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the author of #13018, I want to understand this block better. Seems like we don't need this block or the test/deployer test cases would change with its addition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah; you moved it in this file. You moved it because it was only in use if this function was using it, and because without 'Enabled' it smelled a little funny remaining. There's no point in having the same thing in both places. Looks good.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it would have been dead code here

// Add agentgateway-specific configuration
agwConfig := &kgateway.Agentgateway{
LogLevel: ptr.To("info"),
Image: &kgateway.Image{
Registry: ptr.To(AgentgatewayRegistry),
Tag: ptr.To(AgentgatewayDefaultTag),
Repository: ptr.To(AgentgatewayImage),
PullPolicy: (*corev1.PullPolicy)(ptr.To(imageInfo.PullPolicy)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to default to omitting the pull policy so that k8s can use it's unsurprising logic about 'latest means Always, and otherwise...' (see https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting).

This is confusing and maybe even casts in the wrong place, but also turns the empty string into "something" when I'd prefer it were nil. There's also validation to consider since we're using a string in ImageInfo, not corev1.PullPolicy. Perhaps use this (which will never error because imageInfo is something we control completely):

func ToPullPolicyPtr(input string) (*corev1.PullPolicy, error) {
    policy := corev1.PullPolicy(input)

    switch policy {
    case corev1.PullAlways, corev1.PullIfNotPresent, corev1.PullNever:
        return ptr.To(policy), nil
    case "":
        // let K8s nuanced default apply (see https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting)
        return nil, nil
    default:
        return nil, fmt.Errorf("invalid image pull policy: %q", input)
    }
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm; you just moved this and it's working well enough with a pointer to the empty string because both data plane helm charts say '{{- if $gateway.image.pullPolicy }}'. I don't love the cast on the outside but corev1.PullPolicy is a string like imageInfo.PullPolicy...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW I agree, maybe a followup?

},
}

// Only add SecurityContext if not omitting defaults
if !omitDefaultSecurityContext {
agwConfig.SecurityContext = &corev1.SecurityContext{
AllowPrivilegeEscalation: ptr.To(false),
ReadOnlyRootFilesystem: ptr.To(true),
RunAsNonRoot: ptr.To(true),
RunAsUser: ptr.To[int64](10101),
Capabilities: &corev1.Capabilities{
Drop: []corev1.Capability{"ALL"},
},
}
}

gwp.Spec.Kube.Agentgateway = agwConfig

gwp.Spec.Kube.PodTemplate.ReadinessProbe.HTTPGet.Path = "/healthz/ready"
gwp.Spec.Kube.PodTemplate.ReadinessProbe.HTTPGet.Port = intstr.FromInt(15021)
gwp.Spec.Kube.PodTemplate.StartupProbe.HTTPGet.Path = "/healthz/ready"
Expand Down Expand Up @@ -258,31 +284,13 @@ func defaultGatewayParameters(imageInfo *ImageInfo, omitDefaultSecurityContext b
IstioMetaClusterId: ptr.To("Kubernetes"),
},
},
Agentgateway: &kgateway.Agentgateway{
Enabled: ptr.To(false),
LogLevel: ptr.To("info"),
Image: &kgateway.Image{
Registry: ptr.To(AgentgatewayRegistry),
Tag: ptr.To(AgentgatewayDefaultTag),
Repository: ptr.To(AgentgatewayImage),
PullPolicy: (*corev1.PullPolicy)(ptr.To(imageInfo.PullPolicy)),
},
SecurityContext: &corev1.SecurityContext{
AllowPrivilegeEscalation: ptr.To(false),
ReadOnlyRootFilesystem: ptr.To(true),
RunAsNonRoot: ptr.To(true),
RunAsUser: ptr.To[int64](10101),
Capabilities: &corev1.Capabilities{
Drop: []corev1.Capability{"ALL"},
},
},
},
// Note: Agentgateway config is only added for agentgateway controller gateways
// via defaultAgentgatewayParameters(). For envoy gateways, we leave this nil.
},
},
}
if omitDefaultSecurityContext {
gwp.Spec.Kube.EnvoyContainer.SecurityContext = nil
gwp.Spec.Kube.Agentgateway.SecurityContext = nil
}
return gwp.DeepCopy()
}
Loading