Unlike Travis CI and AppVeyor, GitHub Actions doesn't provide a way to clear cache. In such a situation, it seems it's a common practice to add a version prefix to the key as Circle CI suggests so:
Tip: Caches are immutable so it is useful to start all your cache keys with a version prefix, for example v1-.... This enables you to regenerate all of your caches by incrementing the version in this prefix.
(https://circleci.com/docs/2.0/caching/#clearing-cache)
I followed this on ggplot2: tidyverse/ggplot2@644c436