Allow identity blocks in sidecar_task blocks

[seanamos]

Proposal

Allow the use of identity blocks in sidecar_task blocks.

This is similar in nature to #17987 .

Use-cases

Allow the injection of AWS identities into terminating gateways that invoke lambdas.

Attempted Solutions

It is possible the suggestion at #17987 (comment) would also work for this, that is to construct the job json in it's entirety manually and submit it. I have not tested this yet.

An easier to implement workaround is to add permissions to the EC2 IAM role/policy, but this is undesirable.

If #17987 were implemented and one has a Vault AWS secret backend, that could also work as an alternative solution.

[tgross]

Hi @seanamos! This seems like a reasonable idea. I'll mark it for roadmapping, but I'm not sure when we'd get to this. You may want to look into deploying the gateway job as a standalone task, similar to what we've demonstrated in https://github.com/hashicorp-guides/consul-api-gateway-on-nomad

[3nprob]

We have a different use-case but would also greatly benefit from this. We want to ensure jobs don't break in the existing cluster before we can fully migrating to Workload Identities. In an attempt at safe migration to Workload Identities, not being able to set identity for sidecar_task becomes a blocker: Non-connect jobs can opt-in to WI prior to cluster-wide enforcement to WI by setting identity blocks on any relevant service or task.

Attempting to set identity for a Consul Connect service results in sidecar tasks failing to start due to consul service lookups failing with 403 ACL Not Found. It seems the only way to make Connect services work with Consul WI would be to configure task_identity on Nomad servers: A catch-22.

[3nprob]

• Implemented in feat(api/consul): allow configuring identities for sidecar_task #25877

[tgross]

Attempting to set identity for a Consul Connect service results in sidecar tasks failing to start due to consul service lookups failing with 403 ACL Not Found. It seems the only way to make Connect services work with Consul WI would be to configure task_identity on Nomad servers: A catch-22.

This shouldn't be the case (and is tested end-to-end on a nightly basis). Can you share the job spec, configuration, logs, etc. that are leading you to this conclusion?

[3nprob]

@tgross To clarify, this is on a cluster with no service_identity or task_identity configured, so workloads are by default not opted-in to WI. What would expected behavior here be (and where can we find the test-case?)? Should the sidecar task default to acquiring the service identity if unspecified? Or should it keep using agent token as pre-WI?

Sample job variations below.

🟢 Precondition (job schedules without WI)

:

job "foo" {
  datacenters = ["dc1"]
  group "foo" {
    count = 1
    network {
      mode = "bridge"
      port "http" {
        to = 1234
      }
    }

    service {
      name = "foo-bogus"
      port = "666"
      provider = "consul"
      tags = ["connect"]
      connect {
        sidecar_service {
          disable_default_tcp_check = true
          proxy {
            upstreams {
              destination_name = "bar"
              local_bind_port  = 4567
            }
          }
        }
      }
    }

    service {
      name = "foo-http"
      port = "1234"
      provider = "consul"
      tags = ["connect"]
      connect {
        sidecar_service {
        }
      }
    }

    task "foo" {
      driver = "docker"
      config {
        image = "localhost/foo:latest"
      }
      resources {
        cpu = 100
        memory = 100
      }
    }
    update {
      max_parallel = 1
      health_check = "checks"
    }
  }
}

🔴 Enabling WI for services fails scheduling as described in issue

job "foo" {
  datacenters = ["dc1"]
  group "foo" {
    count = 1
    network {
      mode = "bridge"
      port "http" {
        to = 1234
      }
    }

    consul {}

    service {
      identity {
        name = "default"
        aud = ["consul.io"]
      }
      name = "foo-bogus"
      port = "666"
      provider = "consul"
      tags = ["connect"]
      connect {
        sidecar_service {
          disable_default_tcp_check = true
          proxy {
            upstreams {
              destination_name = "bar"
              local_bind_port  = 4567
            }
          }
        }
      }
    }

    service {
      identity {
        name = "default"
        aud = ["consul.io"]
      }
      name = "foo-http"
      port = "1234"
      provider = "consul"
      tags = ["connect"]
      connect {
        sidecar_service {
        }
      }
    }

    task "foo" {
      driver = "docker"
      config {
        image = "localhost/foo:latest"
      }
      resources {
        cpu = 100
        memory = 100
      }
    }
    update {
      max_parallel = 1
      health_check = "checks"
    }
  }
}

🟡 Non-connect jobs schedule and have their services registered, though

job "baz" {
  datacenters = ["dc1"]
  group "baz" {
    count = 1
    network {
      mode = "bridge"
      port "http" {
        to = 777
      }
    }

    consul {}

    service {
      identity {
        name = "default"
        aud = ["consul.io"]
      }
      name = "baz-http"
      port = "777"
      provider = "consul"
    }

    task "baz" {
      driver = "docker"
      config {
        image = "localhost/baz:latest"
      }
      resources {
        cpu = 100
        memory = 100
      }
    }
    update {
      max_parallel = 1
      health_check = "checks"
    }
  }
}

🟢 With #25877 applied and `sidecar_task.identity` added, connect job schedules again

job "foo" {
  datacenters = ["dc1"]
  group "foo" {
    count = 1
    network {
      mode = "bridge"
      port "http" {
        to = 1234
      }
    }

    consul {}

    service {
      identity {
        name = "default"
        aud = ["consul.io"]
      }
      name = "foo-bogus"
      port = "666"
      provider = "consul"
      tags = ["connect"]
      connect {
        sidecar_service {
          disable_default_tcp_check = true
          proxy {
            upstreams {
              destination_name = "bar"
              local_bind_port  = 4567
            }
          }
        }
        sidecar_task {
          identity {
            name = "default"
            aud = ["consul.io"]
          }
        }
      }
    }

    service {
      identity {
        name = "default"
        aud = ["consul.io"]
      }
      name = "foo-http"
      port = "1234"
      provider = "consul"
      tags = ["connect"]
      connect {
        sidecar_service {
        }
        sidecar_task {
          identity {
            name = "default"
            aud = ["consul.io"]
          }
        }
      }
    }

    task "foo" {
      driver = "docker"
      config {
        image = "localhost/foo:latest"
      }
      resources {
        cpu = 100
        memory = 100
      }
    }
    update {
      max_parallel = 1
      health_check = "checks"
    }
  }
}

Servers are 1.9.7. Client binaries are built based on 1.8.4 with security/bug fixes backported. Consul v1.20.5. All nodes Linux.