Update Warehouse documentation to inform about BigQuery datasets (pypi#8240)

* Update documentation to inform about BigQuery datasets

* Fix linter errors

* Rename rst file

Co-authored-by: Dustin Ingram <[email protected]>

Author: patelneel55

docs/api-reference/bigquery-datasets.rst

@@ -0,0 +1,26 @@
+BigQuery Datasets
+=================
+
+We use BigQuery to serve our public datasets. PyPI offers two tables whose
+data is sourced from projects on PyPI. The tables and its pertaining data are licensed
+under the `Creative Commons License <https://creativecommons.org/licenses/by/4.0/>`_.
+
+Download Statistics Table
+-------------------------
+
+The download statistics table allows you learn more about downloads patterns of
+packages hosted on PyPI. This table is populated through the `Linehaul
+project <https://github.com/pypa/linehaul>`_ by streaming download logs from PyPI
+to BigQuery. For more information on analyzing PyPI package downloads, see the `Python
+Package Guide <https://packaging.python.org/guides/analyzing-pypi-package-downloads/>`_
+
+Project Metadata Table
+----------------------
+
+We also have a table that provides access to distribution metadata
+as outlined by the `core metadata specifications <https://packaging.python.org/specifications/core-metadata/>`_.
+The table is meant to be a data dump of metadata from every
+release on PyPI, which means that the rows in this BigQuery table
+are immutable and are not removed even if a release or project is deleted.
+This data can be accessible under the ``the-psf.pypi.distribution_metadata``
+public dataset on BigQuery.

docs/api-reference/index.rst

@@ -56,8 +56,8 @@ use our RSS feeds.
 No new integrations should use the XML-RPC APIs as they are planned for
 deprecation. Existing consumers should migrate to JSON/RSS/Legacy APIs.
 
-Available APIs
---------------
+Available APIs & Datasets
+-------------------------
 
 .. toctree::
     :maxdepth: 2
@@ -68,3 +68,4 @@ Available APIs
     stats
     xml-rpc
     integration-guide
+    bigquery-datasets