Skip to content

feat: Fix and refactor setup-dev-cluster verification script#522

Merged
droot merged 1 commit into
GoogleCloudPlatform:mainfrom
noahlwest:fix-dev-cluster-eval
Sep 8, 2025
Merged

feat: Fix and refactor setup-dev-cluster verification script#522
droot merged 1 commit into
GoogleCloudPlatform:mainfrom
noahlwest:fix-dev-cluster-eval

Conversation

@noahlwest
Copy link
Copy Markdown
Collaborator

I noticed that this eval would consistently fail when checking for quotas, despite things being set up correctly. This was caused by a small error in the check syntax. This verify script is kind of big, so I also tried to refactor it into something more readable and easy to debug.

  • breaks down checks into functions for each verification step, with a main function orchestrating the checks.
  • adds a cleanup function that runs on exit to ensure all created test resources are removed, even if the script fails.
  • reduces kubectl calls by batch fetching
  • uses the --quiet flag for cleaner output instead of sending to /dev/null
  • uses set -euo pipefail for more predictable behavior on errors.

A successful run looks like this:

Running command: .../kubectl-ai/k8s-bench/tasks/setup-dev-cluster/verify.sh
Starting comprehensive verification of dev cluster setup...
Checking namespaces...
All namespaces exist.
Checking service accounts...
All developer ServiceAccounts exist.
Testing RBAC permissions...
  - alice has full access to their own namespace
  - alice has read access to 'dev-shared'
  - alice is properly isolated from other dev, staging, and prod namespaces
  - bob has full access to their own namespace
  - bob has read access to 'dev-shared'
  - bob is properly isolated from other dev, staging, and prod namespaces
  - charlie has full access to their own namespace
  - charlie has read access to 'dev-shared'
  - charlie is properly isolated from other dev, staging, and prod namespaces
RBAC permissions are correctly configured.
Checking resource quotas...
Resource quotas are correctly configured.
Checking for existence of Network Policies...
At least one NetworkPolicy exists in all namespaces.
Testing network isolation...
  - Creating test pods and services...
pod/test-pod-alice created
pod/test-pod-bob created
pod/test-pod-charlie created
service/test-service-charlie created
  - Waiting for test pods to be ready...
pod/test-pod-alice condition met
pod/test-pod-bob condition met
pod/test-pod-charlie condition met
  - Testing cross-namespace isolation (alice -> bob)...
  - Cross-namespace access is properly blocked.
  - Testing DNS access...
  - DNS access is working correctly.
Network policies are functioning correctly.
All verifications passed! Cluster setup is correctly configured.
Cleaning up test resources...
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "dev-shared" force deleted
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "dev-bob" force deleted
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "prod" force deleted
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "staging" force deleted
namespace "dev-alice" force deleted
namespace "dev-charlie" force deleted
Cleanup complete.

Running command: .../kubectl-ai/k8s-bench/tasks/setup-dev-cluster/cleanup.sh
Cleaning up dev cluster eval resources...
Cleanup completed

Evaluation Results:
==================

Task: setup-dev-cluster
  LLM Config: {ID:shim_disabled-gemini-gemini-2.5-pro ProviderID:gemini ModelID:gemini-2.5-pro EnableToolUseShim:false Quiet:true}
    success

@droot droot merged commit 7bd0cb2 into GoogleCloudPlatform:main Sep 8, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants