Provider crash when adding keycloak_default_roles on new realm

[TBeijen]

TL;DR:

• Starting from Keycloak v22, GET-ing a realm with same access token used to create it, returns an incomplete 200 instead of 403.

  • As was the case in v21 and prior
  • Since provider acceptance test currently don't test v22 upwards, this has gone unnoticed so far
  • This looks like a bug in Keycloak: # Realm GET after POST returns incomplete 200 response instead of 403 (starting from v22) keycloak#26301

• Because of that, not receiving a 403, the roken refresh is no longer triggered. This part (which explicitly mentions the 403 after realm create scenario): https://github.com/mrparkers/terraform-provider-keycloak/blob/master/keycloak/keycloak_client.go#L338

• Since this behaviour change is present in Keycloak versions 22...23.0.4 it's probably needed to address this.

---

Having keycloak_default_roles on a newly created realm causes a provider crash. If the realm already exists all works fine.

Sample code:

resource "keycloak_realm" "sample" {
  realm = "tf_foo"
}

resource "keycloak_default_roles" "sample" {
  # Commenting this out on first apply and all works fine on second apply after uncommenting.
  realm_id      = keycloak_realm.sample.id
  default_roles = ["uma_authorization"]
}

Crash:

keycloak_default_roles.sample: Creating...
╷
│ Error: Plugin did not respond
│
│   with keycloak_default_roles.sample,
│   on main.tf line 47, in resource "keycloak_default_roles" "sample":
│   47: resource "keycloak_default_roles" "sample" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-keycloak_v4.3.1 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x102744754]

goroutine 11 [running]:
github.com/mrparkers/terraform-provider-keycloak/provider.resourceKeycloakDefaultRolesReconcile({0x102ac1808, 0x140003b3560}, 0x0?, {0x102ab3140?, 0x14000380a20?})
  github.com/mrparkers/terraform-provider-keycloak/provider/resource_keycloak_default_roles.go:106 +0x174
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0x140003a7260, {0x102ac1840, 0x1400032d5f0}, 0xd?, {0x102ab3140, 0x14000380a20})
  github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:707 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0x140003a7260, {0x102ac1840, 0x1400032d5f0}, 0x1400019d2b0, 0x1400017f400, {0x102ab3140, 0x14000380a20})
  github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:837 +0x86c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0x140003841b0, {0x102ac1840?, 0x1400032d4d0?}, 0x140005b5e00)
  github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:1021 +0xb70
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0x140001225a0, {0x102ac1840?, 0x1400032cc60?}, 0x14000636850)
  github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:818 +0x3b8
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x102a84000?, 0x140001225a0}, {0x102ac1840, 0x1400032cc60}, 0x140006367e0, 0x0)
  github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0x14000510000, {0x102ac53a0, 0x14000340820}, 0x1400039f680, 0x140000a6900, 0x102fbeb40, 0x0)
  google.golang.org/grpc@v1.51.0/server.go:1340 +0xb7c
google.golang.org/grpc.(*Server).handleStream(0x14000510000, {0x102ac53a0, 0x14000340820}, 0x1400039f680, 0x0)
  google.golang.org/grpc@v1.51.0/server.go:1713 +0x82c
google.golang.org/grpc.(*Server).serveStreams.func1.2()
  google.golang.org/grpc@v1.51.0/server.go:965 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func1
  google.golang.org/grpc@v1.51.0/server.go:963 +0x290

Error: The terraform-provider-keycloak_v4.3.1 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[TBeijen]

It appears as if there is a caching issue in Keycloak. I'm not sure if that's expected behaviour from Keycloak's part
It's not caching, it's returning an incomplete 200 instead of a 403 by keycloak from v22 upwards: keycloak/keycloak#26301

Keycloak version: v22.0.5. Local setup, single pod.

Below script can be used to reproduce the scenario. As it turns out, bound to the access token used, GET-ing the realm using same access token used to create it, returns a minimal response.

After fetching a new access token, the full realm JSON is returned.

Test script and output

Shell script creating, then repeatedly fetching the realm:

#!/bin/sh
set -e

kc_host=$KC_HOST
kc_client_id=$KC_CLIENT_ID
kc_client_secret=$KC_CLIENT_SECRET

realm=$1

get_token() {
    token=$(curl -X POST "$kc_host/realms/master/protocol/openid-connect/token" --http1.1 \
    -d "client_id=$kc_client_id" \
    -d "client_secret=$kc_client_secret" \
    -d 'grant_type=client_credentials' \
    -k -s | jq -r '.access_token')
    echo $token
}

create_realm()  {
    token=$1
    response=$(curl -X POST "$kc_host/admin/realms" --http1.1 \
    -H "Authorization: Bearer $token" \
    -H "Content-Type: application/json" \
    -d '{"realm":"'$realm'","enabled":true}' \
    -k -s)
    echo $response
}

get_realm() {
    token=$1
    qs=$2
    response=$(curl -X GET "$kc_host/admin/realms/${realm}${qs}" --http1.1 \
    -H "Authorization: Bearer $token" \
    -k -s)
    echo $response
}

delete_realm() {
    token=$1
    response=$(curl -X DELETE "$kc_host/admin/realms/${realm}" --http1.1 \
    -H "Authorization: Bearer $token" \
    -k -s)
    echo $response
}

parse() {
    cat |jq -M '{"realm": .realm, "defaultRole.id": .defaultRole.id }'
}

echo
echo "Creating realm $realm"
token=$(get_token)
create_realm $token
echo
echo "Getting realm $realm"
echo $(get_realm $token |parse)
echo 
echo "Sleep & getting realm"
sleep 2
echo $(get_realm $token |parse)
echo 
echo "Cache busting query string & getting realm"
random_string=$(xxd -l4 -ps /dev/urandom)
echo $(get_realm $token "?random=${random_string}" |parse)
echo 
echo "New access token & getting realm"
token=$(get_token)
echo $(get_realm $token |parse)
echo 
echo "Cache busting query string & getting realm"
random_string=$(xxd -l4 -ps /dev/urandom)
echo $(get_realm $token "?random=${random_string}" |parse)
echo
echo "Deleting realm"
delete_realm $token

Sample output:

Creating realm foobar


Getting realm foobar
{ "realm": "foobar", "defaultRole.id": null }

Sleep & getting realm
{ "realm": "foobar", "defaultRole.id": null }

Cache busting query string & getting realm
{ "realm": "foobar", "defaultRole.id": null }

New access token & getting realm
{ "realm": "foobar", "defaultRole.id": "8c1d35b3-6f1b-426d-85a2-e93dd06719d4" }

Cache busting query string & getting realm
{ "realm": "foobar", "defaultRole.id": "8c1d35b3-6f1b-426d-85a2-e93dd06719d4" }

Deleting realm

[vincent-petit]

Hello,
We got the same issue.
As workaround, we notice that if we configured the env var TF_LOG_PATH there is no error after the ealm creation

I saw corrections in the PRs and it seems good. There is a way to include it in the next patchs version of the provider?
Regards,
Vincent