Public database of general project metadata

[Melykuti]

Similarly to how the Linehaul project streams PyPI download data into a Google BigQuery public dataset (into tables called `the-psf:pypi.downloadsYYYYMMDD`), I wish to be able to query a public dataset that contains the project metadata from PyPI, for example how it is presented by Warehouse JSON API calls.

I would like to analyse project licenses. In fact, I would not be the only one interested in this data to do a study. This data is technically simpler to collect than the downloads data and would be similarly informative about the totality of Python packages. I'd be thankful if such a database were created.

[pradyunsg]

Relevant here: https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies/

Note that license information is part of the same "metadata" as dependency information.

[KOLANICH]

I would like to analyse project licenses.

I don't think that copyright trolls (including GPL trolls) should be gifted with a tool for their needs. Of course they can create an own tool, but I don't think they should be helped.

Implementing new specifications like PEP 517

They have forgotten to notice setup.cfg.

Also IMHO using wheels should be encouraged. How about banning all uploads to pypy that are not wheels or their signatures?

[ncoghlan]

Tidelift provide such a database at libraries.io: https://libraries.io/languages/Python

The relevant libraries.io page is linked from the "Statistics" section on each PyPI project page.

There aren't any plans to build a Python specific version of this dataset (since a language independent one is generally more useful).

[ChillarAnand]

Does tidelift dump has all the metadata provided by PyPi or does it provide only a subset of the metadata?