Improve per-tenant alert generator URL template (#7458)

Author: CharlieTLe

CHANGELOG.md

@@ -5,7 +5,7 @@
 * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
 * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481
 * [FEATURE] Ingester: Add experimental active series tracker that counts active series by configurable label matchers (including regex) per tenant and exposes `cortex_ingester_active_series_per_tracker` metric. Configured via `active_series_trackers` in runtime config overrides. #7476
-* [FEATURE] Ruler: Add per-tenant `ruler_alert_generator_url_template` runtime config option to customize alert generator URLs using Go templates. Supports Grafana Explore, Perses, and other UIs. #7302
+* [FEATURE] Ruler: Add per-tenant `ruler_alert_generator_url_template` runtime config option to customize alert generator URLs using Go templates. Includes a `jsonEscape` template function for safely embedding expressions in JSON-encoded URL parameters (e.g., Grafana Explore panes). Supports Grafana Explore, Perses, and other UIs. #7302
 * [FEATURE] Distributor: Add experimental `-distributor.enable-start-timestamp` flag for Prometheus Remote Write 2.0. When enabled, `StartTimestamp (ST)` is ingested. #7371
 * [FEATURE] Memberlist: Add `-memberlist.cluster-label` and `-memberlist.cluster-label-verification-disabled` to prevent accidental cross-cluster gossip joins and support rolling label rollout. #7385
 * [FEATURE] Querier: Add timeout classification to classify query timeouts as 4XX (user error) or 5XX (system error) based on phase timing. When enabled, queries that spend most of their time in PromQL evaluation return `422 Unprocessable Entity` instead of `503 Service Unavailable`. #7374

docs/configuration/config-file-reference.md

@@ -4409,8 +4409,9 @@ query_rejection:
 
 # Go text/template for alert generator URLs. Available variables: .ExternalURL
 # (resolved external URL) and .Expression (PromQL expression). Built-in
-# functions like urlquery are available. If empty, uses default Prometheus
-# /graph format.
+# functions like urlquery are available. A jsonEscape function is also provided
+# for embedding expressions inside JSON-encoded URL parameters. If empty, uses
+# default Prometheus /graph format.
 [ruler_alert_generator_url_template: <string> | default = ""]
 
 # Enable to allow rules to be evaluated with data from a single zone, if other

docs/getting-started/runtime-config.yaml

@@ -11,15 +11,15 @@ overrides:
   tenant-a:
     ruler_external_url: "http://localhost:3000"
     ruler_alert_generator_url_template: >-
-      {{ .ExternalURL }}/explore?schemaVersion=1&panes=%7B%22default%22:%7B%22datasource%22:%22tenant-a%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22{{ urlquery .Expression }}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1
+      {{ .ExternalURL }}/explore?schemaVersion=1&panes=%7B%22default%22:%7B%22datasource%22:%22tenant-a%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22{{ urlquery (jsonEscape .Expression) }}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1
 
   # Tenant using Perses for alert generator URLs.
   # Clicking "Source" on an alert opens Perses explore view with
   # the PromQL expression pre-filled and the TenantB datasource selected.
   tenant-b:
     ruler_external_url: http://localhost:8080
     ruler_alert_generator_url_template: >-
-      {{ .ExternalURL }}/explore?explorer=Prometheus-PrometheusExplorer&data=%7B%22tab%22%3A%22graph%22%2C%22queries%22%3A%5B%7B%22kind%22%3A%22TimeSeriesQuery%22%2C%22spec%22%3A%7B%22plugin%22%3A%7B%22kind%22%3A%22PrometheusTimeSeriesQuery%22%2C%22spec%22%3A%7B%22datasource%22%3A%7B%22kind%22%3A%22PrometheusDatasource%22%2C%22name%22%3A%22tenantb%22%7D%2C%22query%22%3A%22{{ urlquery .Expression }}%22%7D%7D%7D%7D%5D%7D
+      {{ .ExternalURL }}/explore?explorer=Prometheus-PrometheusExplorer&data=%7B%22tab%22%3A%22graph%22%2C%22queries%22%3A%5B%7B%22kind%22%3A%22TimeSeriesQuery%22%2C%22spec%22%3A%7B%22plugin%22%3A%7B%22kind%22%3A%22PrometheusTimeSeriesQuery%22%2C%22spec%22%3A%7B%22datasource%22%3A%7B%22kind%22%3A%22PrometheusDatasource%22%2C%22name%22%3A%22tenantb%22%7D%2C%22query%22%3A%22{{ urlquery (jsonEscape .Expression) }}%22%7D%7D%7D%7D%5D%7D
 
   # Tenants without overrides use the global ruler.external.url
   # and the default Prometheus /graph format.

docs/getting-started/single-binary.md

@@ -228,15 +228,22 @@ The `ruler_alert_generator_url_template` field accepts a Go template with two va
 - `{{ .ExternalURL }}` — the resolved external URL for this tenant (set via `ruler_external_url`)
 - `{{ .Expression }}` — the PromQL expression that triggered the alert
 
-Built-in Go template functions like `urlquery` are available for URL encoding.
+Built-in Go template functions like `urlquery` are available for URL encoding. Cortex also provides a `jsonEscape` function that escapes a string for embedding inside a JSON string value (e.g., `"` → `\"`). Use `jsonEscape` when the expression is placed inside a JSON-encoded URL parameter, such as Grafana's `panes`.
 
-Example for Grafana Explore:
+Example for Grafana Explore (simple query parameter):
 ```yaml
 ruler_external_url: "http://localhost:3000"
 ruler_alert_generator_url_template: >-
   {{ .ExternalURL }}/explore?expr={{ urlquery .Expression }}
 ```
 
+Example for Grafana Explore (JSON-encoded `panes` parameter — use `jsonEscape` to properly escape quotes in expressions):
+```yaml
+ruler_external_url: "http://localhost:3000"
+ruler_alert_generator_url_template: >-
+  {{ .ExternalURL }}/explore?schemaVersion=1&panes=%7B%22default%22:%7B%22datasource%22:%22my-datasource%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22{{ urlquery (jsonEscape .Expression) }}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1
+```
+
 ### Try It Out
 
 1. **Load alertmanager configs** for tenant-a and tenant-b:
@@ -296,6 +303,13 @@ rules:
       severity: critical
     annotations:
       summary: "Error rate exceeds 5%"
+  - alert: AlwaysFiringWithQuotes
+    expr: count(up{job!="nonexistent"} or vector(1))
+    for: 0m
+    labels:
+      severity: info
+    annotations:
+      summary: "Demo alert with quotes in expression"
 EOF
 
 # Alert rules for tenant-b
@@ -320,6 +334,13 @@ rules:
       severity: warning
     annotations:
       summary: "P99 latency exceeds 2s"
+  - alert: AlwaysFiringWithQuotes
+    expr: count(up{job!="nonexistent"} or vector(1))
+    for: 0m
+    labels:
+      severity: info
+    annotations:
+      summary: "Demo alert with quotes in expression"
 EOF
 ```
 

pkg/ruler/ruler.go

@@ -3,6 +3,7 @@ package ruler
 import (
 	"bytes"
 	"context"
+	"encoding/json"
 	"flag"
 	"fmt"
 	"hash/fnv"
@@ -539,6 +540,20 @@ type generatorURLTemplateData struct {
 	Expression  string
 }
 
+// generatorURLTemplateFuncMap contains custom functions available in generator URL templates.
+//   - jsonEscape: escapes a string for embedding inside a JSON string value (e.g., " → \", \ → \\).
+//     Useful when the expression is placed inside a JSON-encoded URL parameter like Grafana's panes.
+var generatorURLTemplateFuncMap = template.FuncMap{
+	"jsonEscape": func(s string) string {
+		b, err := json.Marshal(s)
+		if err != nil {
+			return s
+		}
+		// json.Marshal wraps the string in quotes; strip them to get just the escaped content.
+		return string(b[1 : len(b)-1])
+	},
+}
+
 // generatorURLTemplateCache caches a parsed text/template keyed on the template string.
 // If the template string changes (e.g., via runtime config), the cache is invalidated.
 type generatorURLTemplateCache struct {
@@ -552,7 +567,7 @@ func (c *generatorURLTemplateCache) getOrParse(tmplStr string) (*template.Templa
 	if c.tmpl != nil && c.tmplStr == tmplStr {
 		return c.tmpl, nil
 	}
-	tmpl, err := template.New("generator_url").Parse(tmplStr)
+	tmpl, err := template.New("generator_url").Funcs(generatorURLTemplateFuncMap).Parse(tmplStr)
 	if err != nil {
 		return nil, err
 	}

pkg/ruler/ruler_test.go

@@ -2833,12 +2833,19 @@ func TestExecuteGeneratorURLTemplate(t *testing.T) {
 			expectErr: true,
 		},
 		{
-			name:        "template with multiple variables",
-			tmplStr:     "{{ .ExternalURL }}/explore?left=%7B%22queries%22:%5B%7B%22expr%22:%22{{ urlquery .Expression }}%22%7D%5D%7D",
+			name:        "template with JSON-encoded panes parameter",
+			tmplStr:     "{{ .ExternalURL }}/explore?left=%7B%22queries%22:%5B%7B%22expr%22:%22{{ urlquery (jsonEscape .Expression) }}%22%7D%5D%7D",
 			externalURL: "http://grafana:3000",
 			expr:        "up",
 			expected:    "http://grafana:3000/explore?left=%7B%22queries%22:%5B%7B%22expr%22:%22up%22%7D%5D%7D",
 		},
+		{
+			name:        "grafana explore template with expression containing double quotes",
+			tmplStr:     `{{ .ExternalURL }}/explore?schemaVersion=1&panes=%7B%22default%22:%7B%22datasource%22:%22tenant-a%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22{{ urlquery (jsonEscape .Expression) }}%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1`,
+			externalURL: "http://localhost:3000",
+			expr:        `count(up{job!="nonexistent"} or vector(1))`,
+			expected:    `http://localhost:3000/explore?schemaVersion=1&panes=%7B%22default%22:%7B%22datasource%22:%22tenant-a%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22count%28up%7Bjob%21%3D%5C%22nonexistent%5C%22%7D+or+vector%281%29%29%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1`,
+		},
 		{
 			name:        "javascript URI scheme is rejected",
 			tmplStr:     "javascript://alert('xss')",

pkg/util/validation/limits.go

@@ -227,7 +227,7 @@ type Limits struct {
 	RulerQueryOffset               model.Duration `yaml:"ruler_query_offset" json:"ruler_query_offset"`
 	RulerExternalLabels            labels.Labels  `yaml:"ruler_external_labels" json:"ruler_external_labels" doc:"nocli|description=external labels for alerting rules"`
 	RulerExternalURL               string         `yaml:"ruler_external_url" json:"ruler_external_url" doc:"nocli|description=Per-tenant external URL for the ruler. If set, it overrides the global -ruler.external.url for this tenant's alert notifications."`
-	RulerAlertGeneratorURLTemplate string         `yaml:"ruler_alert_generator_url_template" json:"ruler_alert_generator_url_template" doc:"nocli|description=Go text/template for alert generator URLs. Available variables: .ExternalURL (resolved external URL) and .Expression (PromQL expression). Built-in functions like urlquery are available. If empty, uses default Prometheus /graph format."`
+	RulerAlertGeneratorURLTemplate string         `yaml:"ruler_alert_generator_url_template" json:"ruler_alert_generator_url_template" doc:"nocli|description=Go text/template for alert generator URLs. Available variables: .ExternalURL (resolved external URL) and .Expression (PromQL expression). Built-in functions like urlquery are available. A jsonEscape function is also provided for embedding expressions inside JSON-encoded URL parameters. If empty, uses default Prometheus /graph format."`
 	RulesPartialData               bool           `yaml:"rules_partial_data" json:"rules_partial_data" doc:"nocli|description=Enable to allow rules to be evaluated with data from a single zone, if other zones are not available.|default=false"`
 
 	// Store-gateway.
@@ -443,7 +443,13 @@ func (l *Limits) Validate(nameValidationScheme model.ValidationScheme, shardByAl
 	}
 
 	if l.RulerAlertGeneratorURLTemplate != "" {
-		if _, err := template.New("").Parse(l.RulerAlertGeneratorURLTemplate); err != nil {
+		// Register custom functions so that templates using them pass validation.
+		// The actual implementations are in the ruler package; these stubs just
+		// allow the parser to accept the function names.
+		funcMap := template.FuncMap{
+			"jsonEscape": func(s string) string { return s },
+		}
+		if _, err := template.New("").Funcs(funcMap).Parse(l.RulerAlertGeneratorURLTemplate); err != nil {
 			return fmt.Errorf("invalid ruler_alert_generator_url_template: %w", err)
 		}
 	}

schemas/cortex-config-schema.json

@@ -5548,7 +5548,7 @@
           "x-format": "duration"
         },
         "ruler_alert_generator_url_template": {
-          "description": "Go text/template for alert generator URLs. Available variables: .ExternalURL (resolved external URL) and .Expression (PromQL expression). Built-in functions like urlquery are available. If empty, uses default Prometheus /graph format.",
+          "description": "Go text/template for alert generator URLs. Available variables: .ExternalURL (resolved external URL) and .Expression (PromQL expression). Built-in functions like urlquery are available. A jsonEscape function is also provided for embedding expressions inside JSON-encoded URL parameters. If empty, uses default Prometheus /graph format.",
           "type": "string"
         },
         "ruler_evaluation_delay_duration": {