Implement caching to get around rate limits for service accounts

[TwiN]

Summary

Currently, rate limits are absurdly low for service accounts (IMO) if you plan on pulling multiple items on every terraform plan/apply.
Caching could help us get around this.

Use cases

If you pull 20 items in your terraform project, this means that you can run terraform plan/apply at most 50 times per 24h period on the 1Password and 1Password Families plan (since they have a limit of 1000 read/write per 24h period).

Proposed solution

Implementing a caching system or making a per-item age-before-refresh configuration may allow us to significantly reduce the number calls.

Is there a workaround to accomplish this today?

None that I know of.

References & Prior Work

N/A

[SimonBarendse]

Hi @TwiN , is age-before-refresh a Terraform fundamental or established practice? I did a quick search and didn't find other mentions of this, but could be very well that I've overlooked things. Ideally we'd follow established practices for caching, so that behavior is intuitive and in line what users have come to expect based on their experience with other providers, and so we don't run into incompatibilities with how Terraform is designed to be used. Is the age-before-refresh you're referring to an example of such established practice? If you could share any references to Terraform documentation or providers with similar implementations that would help us assess your proposal before moving forward to a contribution to this end.

Possible Alternatives

Terraform Refresh

While doing that search I came across this https://www.hashicorp.com/blog/new-terraform-planning-options-refresh-false-refresh-only-replace and was wondering if -refresh=false and -refresh-only could cover your needs? If I'm understanding correctly -refresh=false would use the TF State as a "cache".

Increased rate-limits

We're currently evaluating (service account) rate-limits and considering adding more options to increase your limits. If you'd be willing to share more about your needs, I believe that could be useful. I'm copying in @carey404 on this issue, who is leading that research.

[TwiN]

@SimonBarendse age-before-refresh isn't an established practice, it was more of just an example of a parameter name.

As for my use case, I deploy my personal cloud infrastructure using terraform, and I used to have a single 1password item per secret variable, but I ended up merging them into items with multiple fields to get around the rate limit. Even then, while debugging my infrastructure, it's not uncommon for me to run terraform plan/apply a few dozens of time in the span of an hour, which makes hitting the rate limit very easy

As for the -refresh=false suggestion, the issue is that I may not want to retrieve the cached version of non-1password terraform resources

[SimonBarendse]

Thanks, that's helpful context! We're planning some changes that will make it easier to manage an environment of multiple secrets, similar to how you've been merging into a single item.

With all secrets into a single item, I'd expect you'd be able to run terraform plan/apply much more often than a few dozens of time before hitting rate-limits. There's a few additional requests when you need to reauth (sessions live for 10mins by default), but even accounting for that I'd expect you'd be able to invoke terraform plan 100+ times. Are you exceeding that during development of a single personal project?

[TwiN]

@SimonBarendse Initially I didn't have all my secrets in a handful of 1password items; I had dozens of individual 1password items as well, so on those occasions where I'd spend the entire day working on that cloud infrastructure personal project I mentioned, I'd get rate limited pretty quickly.

This is much more noticeable when debugging Terraform issues, where you may run plans several times over the span of a few minutes.

Of course now that I've merged all my secrets into just a handful of 1password items, it happens far less often, but it's still something I have to keep in mind every now and then.

To clarify on why I don't just use one 1password item for all my secrets, it's for the principle of least privilege. I have multiple projects, but not all projects need access to all secrets other projects have access to, so sometimes the granularity is necessary unless the user is ok with having multiple 1password items with duplicated secrets (I personally try to avoid duplicates, or you have to update multiple items when you change them)

[onedr0p]

@SimonBarendse There's a related discussion with ESO https://github.com/external-secrets/external-secrets/discussions/4786, are there any update on making the API limits less restrictive?