Higher-level policy validation and replacement strategy

[timflannagan]

The TrafficPolicy is a monolithic API that supports attachment at various points in the routing hierarchy:

• Individual HTTPRoute rule level (either via sectionName on the CR or via ExtensionRef on the rule definition)
• HTTPRoute wide
• Listener & xListenerSet wide
• Gateway wide

Invalid policy attached Gateway-wide or Listener-wide can produce Envoy NACKs (e.g. malformed transformation template). The first two attachment points listed above are already handled. The remaining higher-level scopes are not, and a failure here can block unrelated routes, halt routing updates, and cause a large blast radius across the data plane.

[lgadban]

A similar scenario is present in the 1.x Gloo Edge version of this project.

If an invalid policy is applied at the Virtual Host tier (the level above route) the entire listener is stripped (i.e. dropped entirely) from the resulting Proxy configuration.
This is an even more destructive action than "route replacement" but is necessary as otherwise the config can result in envoy NACK-ing.

When this scenario happens, an error is logged and the VirtualService resource which contains the invalid VirtualHostOption policy has the error reported on its status, e.g.

status:
  statuses:
    gloo-system:
      reason: "2 errors occurred:\n\t* VirtualHost Error: ProcessingError. Reason:
        invalid virtual host [gloo-system_inject-response-header] while processing
        plugin transformation: envoy validation mode output: error initializing configuration
        '/dev/fd/0': Failed to parse request template: Failed to parse body template
        [inja.exception.parser_error] (at 1:21) unexpected '}'\n, error: command \"\"/usr/local/bin/envoy\"
        \"--mode\" \"validate\" \"--config-path\" \"/dev/fd/0\" \"-l\" \"critical\"
        \"--log-format\" \"%v\"\" failed with error: exit status 1\n\t* VirtualHost
        Error: ProcessingError. Reason: invalid virtual host [gloo-system_inject-response-header]
        while processing plugin transformation: envoy validation mode output: error
        initializing configuration '/dev/fd/0': Failed to parse request template:
        Failed to parse body template [inja.exception.parser_error] (at 1:21) unexpected
        '}'\n, error: command \"\"/usr/local/bin/envoy\" \"--mode\" \"validate\" \"--config-path\"
        \"/dev/fd/0\" \"-l\" \"critical\" \"--log-format\" \"%v\"\" failed with error:
        exit status 1\n\n"
      reportedBy: gloo
      state: Rejected
      subresourceStatuses:
        '*v1.Proxy.gateway-proxy_gloo-system':
          reportedBy: gloo
          state: Accepted

Functionally by either dropping the listener or keeping it with a single catch-all route that has been route replaced, we are following the behavior in GE, as long as we have a way to signal this behavior on the status accordingly.

With "simple" route replacement this is straightforward as we can (and already do) report this on the HTTPRoute itself.
For the “higher” attachment scopes we will need to design this.

The straightforward approach would be to follow our pattern with route replacement and mark either the Gateway or Listener as Accepted=false (or another condition) and use a custom reason indicating the gateway/listener was dropped/replaced due to invalid policy and surface the error in the message.

[timflannagan]

Unblocked - needs one pager to get consensus on the right behavior here and outline any implementation unknowns before 2.1 is released.

[timflannagan]

Note to myself. When attaching invalid policy in STRICT mode, the current route replacement test manifests don't have the policy attached to it. This is because the merged policies function skips policies with IR errors when computed the merged policy, and because of this, methods like ComputeRouteConfiguration won't run their plugin pass application method (e.g. ApplyRouteConfigPlugin) specific to that method. However, the STANDARD mode test manifests will have that invalid configuration because the len(pol.Errors) == 0.