Upgrading packages from private repositories fails using cert and client-cert

[settingorange]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Poetry version: 1.1.4

Issue

To configure the cert and client-cert I am using:

poetry config certificates.private-pypi.cert /path/to/ca.crt
poetry config certificates.private-pypi.client-cert /path/to/client.pem

... from the custom CA & TLS instructions.

And from pyproject.toml:

[[tool.poetry.source]]
name = "private-pypi"
url = "https://private-pypi/simple"

When doing a poetry update to get the latest packages, it fails to get packages from private repositories.

It fails with:

Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
 failed: self signed certificate in certificate chain (_ssl.c:1091)'))

... I started to debug that error it looked like the cert and CA were simply not provided in the http request. Actually here's what I saw:

A single successful request where the cert and CA were passed:

GET https://private-pypi/simple/<package_name>/

And then 5 failed retries that had the SSLCertVerificationError:

GET https://private-pypi/packages/<package_name>/<package>-py3-none-any.whl#md5=blah

And in those 5 requests it looked like the cert and CA were not provided. So in my CI build I am not able to install <package> which happens to be an upgrade. I believe this started to happen with 1.1.0 as I can force version 1.0.10 and then the CI build will succeed and the package will install successfully.

[dusktreader]

Yes, it's because the certs aren't provided to the second request which actually downloads the archive. I'm posting a pull-request to fix this this morning.

[dusktreader]

Additional context:

The _download_archive method doesn't supply certs for the config. Can verify that the issue is relieved by adding a verify keyword to the request

See lib/poetry/installation/executor.py:616:

    def _download_archive(self, operation, link):  # type: (Operation, Link) -> Path
        response = self._authenticator.request(
            "get", link.url, stream=True, io=self._sections.get(id(operation), self._io),
            # verify="ca.crt",  # If I add this in, the request functions fine
        )
        wheel_size = response.headers.get("content-length")
        operation_message = self.get_operation_message(operation)
        message = "  <fg=blue;options=bold>•</> {message}: <info>Downloading...</>".format(
           message=operation_message,
        )

Without it, I get errors like this:

  • Installing fetch-config (0.4.8): Pending...
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 1.0 seconds.
Retrying HTTP request in 1.5 seconds.
Retrying HTTP request in 2.0 seconds.
  • Installing fetch-config (0.4.8): Failed

  SSLError

  HTTPSConnectionPool(host='---redacted---', port=443): Max retries exceeded with url: /api/v4/projects/1275/packages/pypi/files/fd1d8885b89b3586dfcb7266c5880e06836dcb09a6265a90cc55cf2267f2daa2/fetch_config-0.4.8-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))

[tbennett6421]

+1 for me, experiencing Unknown CA in packet data followed by client FIN.
Poetry version 1.1.4

This is in regards specifically to poetry publish and is blocking our adoption of this tool.

No suitable keyring backend found
No suitable keyring backends were found
Using a plaintext file to store and retrieve credentials
Publishing poetry-demo (0.1.1) to dev_artifactory
 - Uploading poetry-demo-0.1.1.tar.gz FAILED

  Stack trace:

  9  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/clikit/console_application.py:131 in run
      129│             parsed_args = resolved_command.args
      130│
    → 131│             status_code = command.handle(parsed_args, io)
      132│         except KeyboardInterrupt:
      133│             status_code = 1

  8  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/clikit/api/command/command.py:120 in handle
      118│     def handle(self, args, io):  # type: (Args, IO) -> int
      119│         try:
    → 120│             status_code = self._do_handle(args, io)
      121│         except KeyboardInterrupt:
      122│             if io.is_debug():

  7  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/clikit/api/command/command.py:171 in _do_handle
      169│         handler_method = self._config.handler_method
      170│
    → 171│         return getattr(handler, handler_method)(args, io, self)
      172│
      173│     def __repr__(self):  # type: () -> str

  6  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/commands/command.py:92 in wrap_handle
       90│         self._command = command
       91│
    →  92│         return self.handle()
       93│
       94│     def handle(self):  # type: () -> Optional[int]

  5  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/console/commands/publish.py:77 in handle
      75│         )
      76│
    → 77│         publisher.publish(
      78│             self.option("repository"),
      79│             self.option("username"),

  4  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/publisher.py:93 in publish
      91│         )
      92│
    → 93│         self._uploader.upload(
      94│             url,
      95│             cert=cert or get_cert(self._poetry.config, repository_name),

  3  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:119 in upload
      117│
      118│         try:
    → 119│             self._upload(session, url, dry_run)
      120│         finally:
      121│             session.close()

  2  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:205 in _upload
      203│     ):  # type: (requests.Session, str, Optional[bool]) -> None
      204│         try:
    → 205│             self._do_upload(session, url, dry_run)
      206│         except HTTPError as e:
      207│             if (

  1  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:224 in _do_upload
      222│             # TODO: Check existence
      223│
    → 224│             resp = self._upload_file(session, url, file, dry_run)
      225│
      226│             if not dry_run:

  UploadError

  Connection Error: We were unable to connect to the repository, ensure the url is correct and can be reached.

  at ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:282 in _upload_file
      278│                         " - Uploading {0} {1}".format(
      279│                             file.name, "FAILED"
      280│                         )
      281│                     )
    → 282│                 raise UploadError(e)
      283│             finally:
      284│                 self._io.write_line("")
      285│
      286│         return resp

[GooseYArd]

With the addition of a patch I supplied in the PR 3490's comments, this seems to fix the issue that we're experiencing when attempting to download an archive from a private pypi repo that requires client certs for all accesses.

[astro-josh]

Also seeing this issue, cant post anymore info but we are able to pull from private repo using certs and using pip install but poetry fails cert error using same certificates and token as used in the pip install.

[astro-josh]

Any updates on this? Currently blocking our project from upgrading poetry past 1.0.10

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.