Feature Request: Support wildcards in credHelpers config

[jonjohnsonjr]

Problem

Currently, it is only possible to configure a credential helper for an exact registry hostname. I'd like to explore the possibility of adding more expressiveness to the config file to simplify certain scenarios. There are two primary "flavors" of registry namespacing that you often see in the wild.

1. Path-namespaced registries

For example, Docker Hub.

All requests go to the same index.docker.io hostname (well... kind of), so the namespacing is based on the path. Notably, this means that I can only configure one identity for authenticating to Docker Hub, unless I engage in some $DOCKER_CONFIG shenanigans to point to different config files based on what I'm doing.

Possible solution: Allow some path-based matching in the credHelpers field to support something like:

{
  "credHelpers": {
    "index.docker.io/my-personal-repo": "pass",
    "index.docker.io/my-work-org": "paranoidcorp"
  }
}

You can imagine that your employer might want to control access to and rotation of credentials for their public repositories, so they developed docker-credential-paranoidcorp for managing those credentials.

2. Domain-namespaced registries

For example, ECR.

Both your AWS account and the region for a registry are encoded in the domain name. This makes things rather frustrating if you're dealing with a lot of registries. From the amazon-ecr-credential-helper docs, you can see that the first example recommends just setting credsStore:

{
	"credsStore": "ecr-login"
}

Why? Probably because you would otherwise have to configure every registry individually, which would be rather cumbersome. A major drawback of this global credsStore is that it precludes using any other cred helper for any other registries, so if you talk to anything other than ECR, you're out of luck.

It would be wonderful if you could do something like:

{
  "credHelpers": {
    "*.dkr.ecr.*.amazonaws.com": "ecr-login"
  }
}

This would work for all regions and all accounts.

Some registries (e.g. GCR) even suffer from both of these problems, where there is a region encoded in the hostname and the namespace is encoded in the path.

Solutions?

I'm not sold on this exact syntax from my examples, but I think they demonstrate my point pretty well. I tried to find any previous discussion around this, but had no luck.

Is there any reason not to do something like this? I imagine deciding on an appropriate syntax will be rather difficult (to avoid breaking current config files), and we might need to land some changes in the credential helper protocol, but I don't see any technical blockers.

Personally, I'm most affected by the second case, where it would be nice to support wildcards (or something) in the hostname. (I don't really care about path stuff, but it is something to consider.)

If it's simpler to merge, a simple subdomain matching thing would be fantastic, e.g. this would be a huge quality of life improvement for me:

{
  "credHelpers": {
    "gcr.io": "gcr",
    "*.gcr.io": "gcr"
  }
}

Just so I could say gcr.io or any subdomain of gcr.io should use docker-credential-gcr instead of having an individual line for every region.

cc @dekkagaijin @tiborvass @vdemeester

[jonjohnsonjr]

A simple implementation would be to just allow regular expressions here. It's a lot more powerful than we need, but would cover every case without having to think about it too hard.

My regex-fu is terrible, but something like:

{
  "credHelpers": {
    "^(.*\\.)?gcr.io$": "gcr"
  }
}

would be easy enough to detect to drop into a regex mode (if key[0] == "^") and attempt to regexp.Match the registryHostname (roughly, here).

Obviously, this is ugly, getting the escaping right is tricky, and executing regular expressions all the time might hurt perf, but it would work :P

[samuelkarp]

As the author of the ECR credential helper, I'd also be interested in seeing this.

[dekkagaijin]

I had initially suggested a scheme with wildcards, but was rejected at the time. I don't remember if any reason was given. FWIW implementing this support with regex would not be backwards compatible (because .), but a *-based scheme would be:

{
  ...
  "credHelpers": {
    "gcr.io": "gcr",
    "*.gcr.io": "gcr",
    "*-docker.pkg.dev": "gcr",
    "public.ecr.aws": "ecr-login",
    "*.dkr.ecr.*.amazonaws.com": "ecr-login"
  },
  ...
}

[jonjohnsonjr]

backwards compatible

I've also considered that we might want to add a new config section for this, to avoid accidentally breaking existing config files. It might make sense to have a list instead of a map, if we want to have logic for "first regex that matches"...

[OrahKokos]

👍 Yes please.

[andraxin]

Honestly, wildcards and regexes feel unnecessarily complicated.

For an 80+% coverage, a simple while-loop with suffix lookups
should suffice and that's fully backwards compatible with any
existing configurations...

I haven't looked at the actual code, which is likely in Golang,
but this simple Python implementation shows the idea

# re-use existing `lookup_helper` method
lookup_helper_direct = lookup_helper

# override existing `lookup_helper` method
def lookup_helper(login_uri):
    helper = None
    while login_uri:
       helper = lookup_helper_direct(login_uri)
       if helper: break
       login_uri = remove_component(login_uri)
    return helper

def remove_component(uri):
    index = uri.find('.')
    return None if index < 0 else uri[index+1:]

With that approach, the following configuration
should "just" work (no wildcards or regexes!):

{
  ...
  "credHelpers": {
    "gcr.io": "gcr",
    "pkg.dev": "gcr",
    "public.ecr.aws": "ecr-login",
    "amazonaws.com": "ecr-login"
  },
  ...
}

[bruecktech]

for me this config seems to work as a workaround to avoid having to list every AWS account

{
	"credsStore": "ecr-login",
	"credHelpers": {
		"index.docker.io": "desktop"
	}
}

[andraxin]

@bruecktech

for me this config seems to work as a workaround to avoid having to list every AWS account

Right. If all you have are AWS accounts then that approach works.
I don't see how ecr-login will help with Azure, GCP, or any other provider.

[joshuaspence]

I've just started using https://github.com/docker-credential-magic/docker-credential-magic for this and it works quite well