upload fail in middle with `HTTPError: 400; This filename has already been used`

[Borda]

Hello, we have seen the same failed upload for the last two releases. We have built a package as a wheel and source and used pypa/gh-action-pypi-publish@v1.8.7 to upload them to PyPI. First, it uploads the wheel, which passes, and the second source package fails in the middle. Also, this failed in middle yields that we can't re-upload the source package manually with twine.

Uploading distributions to https://upload.pypi.org/legacy/
INFO     dist/torchmetrics-1.0.1-py3-none-any.whl (712.2 KB)                    
INFO     dist/torchmetrics-1.0.1.tar.gz (420.2 KB)                              
INFO     username set by command options                                        
INFO     password set by command options                                        
INFO     username: __token__                                                    
INFO     password: <hidden>                                                     
Uploading torchmetrics-1.0.1-py3-none-any.whl
25l
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/765.4 kB • --:-- • ?
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/765.4 kB • --:-- • ?
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 765.4/765.4 kB • 00:00 • 11.9 MB/s
25hINFO     Response from https://upload.pypi.org/legacy/:                         
         200 OK                                                                 
Uploading torchmetrics-1.0.1.tar.gz
25l
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/449.1 kB • --:-- • ?
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 449.1/449.1 kB • 00:00 • 148.9 MB/s
25hINFO     Response from https://upload.pypi.org/legacy/:                         
         [40](https://github.com/Lightning-AI/torchmetrics/actions/runs/5542425777/jobs/10117097368#step:7:41)0 This filename has already been used, use a different version. See  
         https://pypi.org/help/#file-name-reuse for more information.           
INFO     <html>                                                                 
          <head>                                                                
           <title>400 This filename has already been used, use a different      
         version. See https://pypi.org/help/#file-name-reuse for more           
         information.</title>                                                   
          </head>                                                               
          <body>                                                                
           <h1>400 This filename has already been used, use a different version.
         See https://pypi.org/help/#file-name-reuse for more information.</h1>  
           The server could not comply with the request since it is either      
         malformed or otherwise incorrect.<br/><br/>                            
         This filename has already been used, use a different version. See      
         https://pypi.org/help/#file-name-reuse for more information.           
                                                                                
                                                                                
          </body>                                                               
         </html>                                                                
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         This filename has already been used, use a different version. See      
         https://pypi.org/help/#file-name-reuse for more information.

See full action log:

• 1.0.0 - https://github.com/Lightning-AI/torchmetrics/actions/runs/5462339923/jobs/9941573869
• 1.0.1 - https://github.com/Lightning-AI/torchmetrics/actions/runs/5542425777/jobs/10117097368

[webknjaz]

@Borda this is because files on PyPI are immutable. If you or somebody else uploaded an artifact with the same name in the past, you cannot reuse it. Even if you deleted it, even if it used to be someone else's project that got cleaned up and you got the PyPI project name.
The error isn't coming from the action but from PyPI (via twine). Nothing actionable for us, closing.

[webknjaz]

FWIW if you think there might be a bug, this would need to be reported to Warehouse and/or twine.

cc @di @woodruffw @pquentin twine version has been updated in the action recently — are you aware of any possibility/reports of the upload process misbehaving?

[webknjaz]

@di could you look into this on the PyPI side? There's no tarballs existing on the CDN it seems:

$ curl -v https://files.pythonhosted.org/packages/source/t/torchmetrics/torchmetrics-1.0.0rc1.tar.gz 2>&1 | grep location:  
< location: https://files.pythonhosted.org/packages/29/64/8507eb5eaee27b281ad71b4f4e90b2db50a5088d01f04eae77739a06976d/torchmetrics-1.0.0rc1.tar.gz

$ curl -v https://files.pythonhosted.org/packages/source/t/torchmetrics/torchmetrics-1.0.0.tar.gz 2>&1 | grep location: 

$ curl -v https://files.pythonhosted.org/packages/source/t/torchmetrics/torchmetrics-1.0.1.tar.gz 2>&1 | grep location:

[pquentin]

This is the first such report I'm seeing, but note that this is using v1.8.7 which does not include #168.

[webknjaz]

Ah, for some reason I thought this was the recent version of the action...

[webknjaz]

Their last successful sdist upload was on Jun 29, it was v1.0.0rc1. Our action's v1.8.7 was released on Jun 26. So they're a chance that the working upload was using an older action version. But if not, this would make it a PyPI problem, not twine.

[webknjaz]

In v1.8.7 the cryptography and requests pins were bumped. But since then, somebody else would also face this problem if it was related.

The only Warehouse change that looks like it could be related is pypi/warehouse#14027 — it tightens the sdist filename validation.

[Borda]

This is the first such report I'm seeing, but note that this is using v1.8.7 which does not include #168.

is this bump so critical?

this is because files on PyPI are immutable.

yes we are aware of this

If you or somebody else uploaded an artifact with the same name in the past, you cannot reuse it. Even if you deleted it, even if it used to be someone else's project that got cleaned up and you got the PyPI project name.

I personally created that project name two years ago and was most of the time the solo owner. if we used a version in the past and was removed I believe I would not be able to upload also wheel package, correct?

[webknjaz]

if we used a version in the past and was removed I believe I would not be able to upload also wheel package, correct?

Only if that version had a wheel. If the version in the past only had an sdist published, the problem would only manifest for sdists but not wheels since there's nothing to conflict with.

[Borda]

Only if that version had a wheel. If the version in the past only had an sdist published, the problem would only manifest for sdists but not wheels since there's nothing to conflict with.

interesting and could it be someone who owned the project name in the past, deleted it all and the name become available so could take it without any indication it was used in the past?

[webknjaz]

interesting and could it be someone who owned the project name in the past, deleted it all and the name become available so could take it without any indication it was used in the past?

Exactly. But I'd like to ask @di to verify this assumption. If that's the case, you'll probably hit similar problems in the future.

[pquentin]

This is the first such report I'm seeing, but note that this is using v1.8.7 which does not include #168.

is this bump so critical?

No, but it bumps various packages, including a few major versions bumps so it might have introduced a bug.