Keyring not used when un-substituted env var is in source credentials url, no pleasant way to override source config

[anentropic]

Describe the bug

I have a private CodeArtifact package index, unfortunately this needs a short-lived token for auth.

Locally I can use pdm self add keyring keyrings.codeartifact and omit the auth from the url, keyring manages it and this all works fine.

However I also need to be able to pdm install inside a multi-stage docker build, where keyring is not available.

The docs advise using env var substitution: https://pdm-project.org/latest/usage/config/#store-credentials-with-the-index

Ideally I want one definition that works for both local keyring and Docker. Obvious thing to try is supply the whole credentials part of the url (aws:<token>@...) as a single env var, so something like:

url = "https://${CODEARTIFACT_TOKEN_PREFIX}mydomain-12345678.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple"

I would expect that if CODEARTIFACT_TOKEN_PREFIX is not set (the 'local' case) then this substitutes to nothing and falls back to keyring auth.

But it doesn't, the un-set env var substitution seems to prevent keyring from being tried. When I pdm lock I get:

[CandidateNotFound]: Unable to find candidates for my-private-package. There may exist some issues with the package name or network condition.

If I do CODEARTIFACT_TOKEN_PREFIX="" pdm lock then it works, but this is annoying.

So that is problem number 1.

In trying to work around this I have encountered some other problems:

2. I tried specifying multiple sources:

    [tool.pdm.resolution]
    respect-source-order = true
   
    [[tool.pdm.source]]
    name = "hub"
    url = "https://mydomain.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple"
    verify_ssl = true
   
    [[tool.pdm.source]]
    name = "hub"
    url = "https://${CODEARTIFACT_TOKEN_PREFIX}mydomain.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple"
    verify_ssl = true

   This fails either locally or in Docker depending which source is placed first - if the source returns an error (instead of just package is missing) it doesn't fall-back to the next source.

   Removing respect-source-order = true fixes this but makes locking super slow, I guess because it tries both and fails one of the sources for every package.

3. My next thought was to use the combination of pyproject.toml and pdm.toml config files. The latter is only vaguely documented.

   [[tool.pdm.source]] in pyproject.toml and pypi.<name>.* in pdm.toml are roughly equivalent, though the source def supports more properties like include_packages.

   It'd be great if you could override one with the other. So e.g. I would define [[tool.pdm.source]] for local and then add a pdm.toml in the docker build to override it. But they are separate configs, and whichever one is defined in pyproject.toml will cause the other build environment to fail.

   If I only define pypi.<name>.* in pdm.toml then I can get something working. I thought I could maybe do like this in docker:

    pdm install -C"pypi.myrepo.url=https://${CODEARTIFACT_TOKEN_PREFIX}mydomain.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple"

   ...but this does nothing so it seems that --config-setting CONFIG_SETTING, -C CONFIG_SETTING "Pass options to the builder" refers to something other than pdm config.

   What does work is to use pdm config --local to update the contents of the pdm.toml file, in docker.

4. There is pypi.ignore_stored_index=true to have it ignore pypi sources in pdm.toml...

   For my case I'd also need the inverse - a config to ignore [[tool.pdm.source]] in pyproject.toml

   That way I could have local config in pyproject.toml and docker config in a pdm.toml file when needed

To reproduce

pdm self add keyring keyrings.codeartifact

Add a source like this to pyproject.toml

[[tool.pdm.source]]
name = "hub"
url = "https://${CODEARTIFACT_TOKEN_PREFIX}mydomain-12345678.d.codeartifact.eu-west-1.amazonaws.com/pypi/myrepo/simple"
verify_ssl = true

Add a package dependency from the private index

Without setting the CODEARTIFACT_TOKEN_PREFIX env var, try pdm lock.

Expected Behavior

When CODEARTIFACT_TOKEN_PREFIX is not set I would expect ${CODEARTIFACT_TOKEN_PREFIX} to resolve to empty string, and for keyring to be used.

Environment Information

PDM version:
  2.26.0
Python Interpreter:
  /Users/anentropic/dev/myproject/.venv/bin/python (3.11)
Project Root:
  /Users/anentropic/dev/myproject
Local Packages:
  /Users/anentropic/dev/myproject/.venv/lib/python3.11/site-packages
{
  "implementation_name": "cpython",
  "implementation_version": "3.12.7",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "23.6.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 23.6.0: Mon Jul 29 21:16:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8112",
  "python_full_version": "3.11.10",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "darwin"
}

Verbose Command Output

No response

Additional Context

No response

Are you willing to submit a PR to fix this bug?

• Yes, I would like to submit a PR.