Recovering from an invalid key in Vault by vault-secrets-operator in VaultSecret object

[michalgoldys]

Vault-Secerts-Operator helm chart version: 1.19.1
AWS EKS Version: v1.24

What is happening:
After inserting an invalid value in Vault, for example with additional space (like: "TEST_OBJECT ") VaultSecret object shows an error state (which at this moment is demanded) like:

 Secret "NAME_OF_SECRET" is invalid: data[TEST_OBJECT  ]: Invalid
        value: "TEST_OBJECT ": a valid config key must consist of
        alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or
        'KEY_NAME',  or 'key-name', regex used for validation is
        '[-._a-zA-Z0-9]+')

After fixing the key value in Vault (in that case, remove hidden space) - it doesn't cause the VaultSecret object to be updated/recreated and thus Secret. IMO it should retry, for example, every 1 minute to query the Vault if the value has been fixed.
In that case, only deleting/recreating either VaultSecret or vault-secrets-operator pod solves the problem.

Expected behaviour:
After fixing the invalid syntax value in Vault - the vault-secrets-operator should update VaultSecret and Secret object. It should query periodically if the wrong object is fixed to propagate new values, instead of beginning stuck.

[ricoberger]

Hi @michalgoldys, do you have set the vault.reconciliationTime value in the Helm chart?

[michalgoldys]

Yes, we've set:

reconciliationTime: 30

[ricoberger]

Hi @michalgoldys, I think the problem might be the https://github.com/kubernetes/client-go/blob/2a5f18df73b70cb85c26a3785b06162f3d513cf5/util/workqueue/default_rate_limiters.go#L39 which has a exponential retry mechanism for failed reconciliations.

So if I create a secret with an invalid key and fix it within some seconds the secret will also be applied in the cluster very fast. If I detect the mistake after some minutes, it will also take longer after the secret is fixed in the cluster. The following times should show this:

test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                0s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                1s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                2s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                3s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                6s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                11s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                22s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                42s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                84s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                2m46s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   0s                5m30s
test                                  False       CreateFailed   Secret "test" is invalid: data[TEST_OBJECT ]: Invalid value: "TEST_OBJECT ": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name',  or 'KEY_NAME',  or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+')   1s                10m

Could this be the problem in your case?

I think the best way to fix it, is to always retry failed reconciliations after 1 minute or so.