tilt: improve Tilt development workflow (#12956) #13255

Wolf-06 · 2026-01-09T19:40:16Z

Description

This PR improves the Tilt development workflow by addressing issue #12956.

Changes

Removed default test values file reference: The Tiltfile no longer defaults to using test/kubernetes/e2e/tests/manifests/common-recommendations.yaml, which was meant for CI/E2E testing, not local development.
Added KGW_DISABLE_LEADER_ELECTION via --set: Leader election is now disabled by default when using Tilt, which is required for debugging. This is passed via Helm --set controller.extraEnv.KGW_DISABLE_LEADER_ELECTION='true' instead of a separate values file.
Verified KGW_VALIDATION_MODE=STRICT works: Tested that ValidationMode=strict works correctly with the current setup. Users can enable it by adding a custom values file to helm_values_files in [tilt-settings.yaml]

How to Test

Run tilt up.
Inspect the deployment manifest or logs to confirm KGW_DISABLE_LEADER_ELECTION is set to 'true'.

Closes #12956

Files Modified

Tiltfile
tilt-settings.yaml

Change Type

/kind cleanup

Changelog

NONE

Additional Notes

Remove test values file reference and add KGW_DISABLE_LEADER_ELECTION=true via --set Signed-off-by: Wolf-06 <[email protected]>

yuval-k · 2026-01-12T19:44:44Z

tilt-settings.yaml

-helm_values_files:
- ./test/e2e/tests/manifests/common-recommendations.yaml
+helm_values_files: 
+  - /tmp/strict-mode.yaml


where is this value coming from?

Apparently, I have committed a local file which was meant to test KGW_VALIDATION_MODE=STRICT .
I will clean it.
Edit: fixed it
thank you

Signed-off-by: Wolf-06 <[email protected]>

Copilot

Pull request overview

This PR improves the Tilt development workflow by removing CI-specific configuration and enabling debugger-friendly settings for local development.

Changes:

Removed default reference to test/kubernetes/e2e/tests/manifests/common-recommendations.yaml which was intended for CI/E2E testing
Added --set controller.extraEnv.KGW_DISABLE_LEADER_ELECTION='true' to disable leader election during local development, which is required for debugging
Updated configuration files to use empty helm values by default with helpful comments for customization

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
Tiltfile	Removed test values file reference from default settings and added leader election disabling via helm `--set` flag in both template and install commands
tilt-settings.yaml	Changed helm_values_files from referencing the test file to an empty array with a comment showing how to add custom values files

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

puertomontt · 2026-01-13T00:17:37Z

I don't think validation mode will work until envoyinit Dockerfile is incorporated.

To test if validation mode is working set KGW_VALIDATION_MODE to STRICT.
Then if you apply an invalid policy, e.g. one of the test files

you should NOT see this error in the kgateway controller logs

invalid matcher configuration: validation failed: envoy validate invocation failed: exec: "docker": executable file not found in $PATH

Signed-off-by: Wolf-06 <[email protected]>

Wolf-06 · 2026-01-13T13:58:41Z

Hi @puertomontt @yuval-k
I have made changes and the strict mode works properly.

Changes:

Dynamic Helm Configuration: The Tiltfile was updated to support a new helm_sets configuration, allowing arbitrary inline --set flags to be dynamically injected into the Helm installation commands.
Strict Validation Mode: tilt-settings.yaml now utilizes the new helm_sets feature to configure the controller with KGW_VALIDATION_MODE: "strict" via environment variables.
Custom Envoy Build: A custom dockerfile_contents block was added to tilt-settings.yaml to build a development image containing a specific Envoy binary version (1.36.3-patch1) and debugging tools. (it was referenced from the envoyinit dockerfile)

How to Test Strict Validation Mode

Prerequisites

# Create Kind cluster with local registry
ctlptl create cluster kind --name kind-kgateway --registry=ctlptl-registry

# Build and load images
VERSION=1.0.0-ci1 CLUSTER_NAME=kgateway make kind-build-and-load

Step 1: Configure Validation Mode

Edit tilt-settings.yaml:

helm_sets:
  controller.extraEnv.KGW_VALIDATION_MODE: "strict"  # or "standard"

Step 2: Start Tilt

tilt up

Step 3: Apply Test Resources

# Apply Gateway
kubectl apply -f examples/example-gw.yaml

# Apply invalid route with malformed regex
kubectl apply -f test/e2e/features/routereplacement/testdata/invalid-matcher.yaml

Step 4: Verify Results

# Check HTTPRoute status
kubectl get httproute invalid-matcher-route -n default -o yaml | grep -A 10 "conditions:"

Expected Results:

Mode	`status`	`reason`	`message`
STANDARD	`True`	`Accepted`	"Successfully accepted Route"
STRICT	`False`	`RouteRuleDropped`	"Dropped Rule (0): invalid matcher configuration..."

Step 5: Check Controller Logs

kubectl logs -n kgateway-system deployment/kgateway --tail=20 | grep -i "invalid"

In STRICT mode, you'll see:

{"msg":"invalid matcher","error":"invalid matcher configuration: validation failed: invalid xds configuration: error initializing configuration '/dev/fd/0': missing ]: [a-z"}

Switching Between Modes

Edit tilt-settings.yaml:

helm_sets:
  controller.extraEnv.KGW_VALIDATION_MODE: "standard"  # or "strict"

The controller will pick up the change automatically via Tilt's live reload

Delete and re-apply the HTTPRoute to trigger re-evaluation:

kubectl delete httproute invalid-matcher-route -n default
kubectl apply -f test/e2e/features/routereplacement/testdata/invalid-matcher.yaml

Notes

Default mode is STANDARD if not specified
The custom Dockerfile includes Envoy binary at /usr/local/bin/envoy for STRICT mode

Thank you

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Tiltfile

tilt-settings.yaml

… fixes Signed-off-by: Wolf-06 <[email protected]>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tilt-settings.yaml

Tiltfile

davidjumani · 2026-01-13T17:52:31Z

tilt-settings.yaml

-
 helm_installation_name: kgateway
-helm_values_files:
- ./test/e2e/tests/manifests/common-recommendations.yaml


I'm personally not in favour of this as the common-recommendations.yaml contain recommended values, especially in prod. They are used in tests and tilt is used for validating behaviour when writing tests, not just local dev
Instead, devs can just comment out this line when bringing up tilt

The file is kind of a holdover from v1, doesn't really have much in it. Tilt needs some specific values, esp. KGW_DISABLE_LEADER_ELECTION.

Looking at the file again, I see now that it doesn't have a lot and is leftover from v1, but afaict most tests use this as the ProfileValuesManifestFile. The other specific values can be added to a tilt-profile.yaml or passed as helm flags in tilt-setting.yaml

davidjumani · 2026-01-13T17:54:43Z

tilt-settings.yaml

    label: kgateway
    # Command to build the binary
-    #build_binary: GCFLAGS='all="-N -l"' make -B kgateway
+    #build_binary: make -B kgateway GCFLAGS="all=-N -l" LDFLAGS=""


Could this change be reverted? It only updates a comment

Hi @davidjumani
It is not just a comment. It is an option of which type of binary to build.
make -B kgateway GCFLAGS="all=-N -l" LDFLAGS="" is used to build the kgateway binary specifically for debugging.
I will add a comment to make it more clear, does that work ?
Thank you

What I mean is that the line is commented out, so any changes made don't have an effect. The command to build the binary is the subsequent line build_binary: make -B kgateway

make -B kgateway and make -B kgateway GCFLAGS="all=-N -l" LDFLAGS="" are basically two options for building the binary with different motives.

With make -B kgateway GCFLAGS="all=-N -l" LDFLAGS="" we build binary with motive of debugging

With make -B kgateway we build the normal binary with optimisation and Inlining.

Use case:

use the standard one for normal coding and testing.

switch to the debugger one ONLY if we are actively trying to attach a debugger

We toggle between the options by commenting the other and uncommenting the one we wan to use.
i will add comments for the instruction on using them to make it clear.

Has debugging been verified with this change ?

Yes
I have verified the debugging with the help of breakpoints in reconcile func (in pkg/kgateway/controller and it hit the breakpoints.
I tried the same with normal build (keeping the debug_port as it is), it didn't hit the breakpoint.

davidjumani · 2026-01-13T17:55:06Z

Tiltfile

    "helm_installation_name": "kgateway",
    "helm_installation_namespace": "kgateway-system",
-    "helm_values_files": ["./test/kubernetes/e2e/tests/manifests/common-recommendations.yaml"],
+    "helm_values_files": [],


Not in favour of this change

davidjumani · 2026-01-13T17:56:45Z

Tiltfile

+    """
+    return str(value).replace("'", "'\"'\"'")
+
+get_resources_cmd = "{0} -n {1} template {2} --include-crds install/helm/kgateway/ --set image.pullPolicy='Never' --set image.registry=ghcr.io/kgateway-dev --set image.tag='{3}' --set controller.extraEnv.KGW_DISABLE_LEADER_ELECTION='true'".format(helm_cmd, settings.get("helm_installation_namespace"), settings.get("helm_installation_name"), image_tag)


Lets move this over helm_sets. This should be as clean as possible as it only gets the resources and not installs kgateway

Working on it.

davidjumani · 2026-01-13T17:57:27Z

Tiltfile

            kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || {{ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml; }} ;
            {0} upgrade --install -n {1} --create-namespace kgateway-crds install/helm/kgateway-crds ;
-            {0} upgrade --install -n {1} --create-namespace {2} install/helm/kgateway/ --set controller.image.pullPolicy='Never' --set image.registry=ghcr.io/kgateway-dev --set image.tag='{3}'""".format(helm_cmd, settings.get("helm_installation_namespace"), settings.get("helm_installation_name"), image_tag)
+            {0} upgrade --install -n {1} --create-namespace {2} install/helm/kgateway/ --set controller.image.pullPolicy='Never' --set image.registry=ghcr.io/kgateway-dev --set image.tag='{3}' --set controller.extraEnv.KGW_DISABLE_LEADER_ELECTION='true'""".format(helm_cmd, settings.get("helm_installation_namespace"), settings.get("helm_installation_name"), image_tag)


as above, lets move this over to helm_sets

Working on it.

davidjumani · 2026-01-13T20:49:31Z

tilt-settings.yaml

 helm_installation_namespace: kgateway-system

+# Inline Helm --set flags (key: value format)
+helm_sets: {}


can this be renamed to helm_flags

Signed-off-by: Wolf-06 <[email protected]>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Tiltfile

tilt-settings.yaml

Tiltfile

tilt: improve Tilt development workflow (kgateway-dev#12956)

d6f0847

Remove test values file reference and add KGW_DISABLE_LEADER_ELECTION=true via --set Signed-off-by: Wolf-06 <[email protected]>

yuval-k reviewed Jan 12, 2026

View reviewed changes

chore: revert local test values in tilt-settings

1a4ef53

Signed-off-by: Wolf-06 <[email protected]>

Copilot AI review requested due to automatic review settings January 12, 2026 20:51

Copilot started reviewing on behalf of Wolf-06 January 12, 2026 20:51 View session

Copilot AI reviewed Jan 12, 2026

View reviewed changes

yuval-k enabled auto-merge January 12, 2026 22:24

Add helm_sets support and Envoy binary for strict validation mode

c958521

Signed-off-by: Wolf-06 <[email protected]>

auto-merge was automatically disabled January 13, 2026 13:58
Head branch was pushed to by a user without write access

Wolf-06 requested a review from Copilot January 13, 2026 14:00

Copilot started reviewing on behalf of Wolf-06 January 13, 2026 14:00 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

Tiltfile Outdated Show resolved Hide resolved

Tiltfile Outdated Show resolved Hide resolved

tilt-settings.yaml Outdated Show resolved Hide resolved

Wolf-06 force-pushed the tilt-improvement branch from 1a3ea70 to b111c5f Compare January 13, 2026 15:25

feat: enhance Tilt setup with strict mode debugging and shell quoting…

ead187c

… fixes Signed-off-by: Wolf-06 <[email protected]>

Wolf-06 force-pushed the tilt-improvement branch from b111c5f to ead187c Compare January 13, 2026 15:28

Wolf-06 requested a review from Copilot January 13, 2026 15:29

Copilot started reviewing on behalf of Wolf-06 January 13, 2026 15:29 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

tilt-settings.yaml Show resolved Hide resolved

Tiltfile Show resolved Hide resolved

davidjumani reviewed Jan 13, 2026

View reviewed changes

feat: refactor tilt helm flags and revert common recommendations config

65bcaee

Signed-off-by: Wolf-06 <[email protected]>

Wolf-06 requested a review from Copilot January 14, 2026 20:51

Wolf-06 requested review from davidjumani and puertomontt January 14, 2026 20:51

Copilot started reviewing on behalf of Wolf-06 January 14, 2026 20:51 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Tiltfile Show resolved Hide resolved

Tiltfile Show resolved Hide resolved

tilt-settings.yaml Show resolved Hide resolved

Tiltfile Show resolved Hide resolved

tilt: improve Tilt development workflow (#12956) #13255

Are you sure you want to change the base?

tilt: improve Tilt development workflow (#12956) #13255

Uh oh!

Conversation

Wolf-06 commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

How to Test

Files Modified

Change Type

Changelog

Additional Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wolf-06 Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

puertomontt commented Jan 13, 2026

Uh oh!

Wolf-06 commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes:

How to Test Strict Validation Mode

Prerequisites

Step 1: Configure Validation Mode

Step 2: Start Tilt

Step 3: Apply Test Resources

Step 4: Verify Results

Step 5: Check Controller Logs

Switching Between Modes

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

davidjumani Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

puertomontt Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidjumani Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wolf-06 Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wolf-06 Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Use case:

Uh oh!

Choose a reason for hiding this comment

Wolf-06 commented Jan 9, 2026 •

edited

Loading

Wolf-06 Jan 12, 2026 •

edited

Loading

Wolf-06 commented Jan 13, 2026 •

edited

Loading

davidjumani Jan 13, 2026 •

edited

Loading

puertomontt Jan 13, 2026 •

edited

Loading

davidjumani Jan 13, 2026 •

edited

Loading

Wolf-06 Jan 13, 2026 •

edited

Loading

Wolf-06 Jan 13, 2026 •

edited

Loading