[TRACKER]: Add dataset cards to uploaded datasets

[Smith42]

Once we have the dataset uploaded to Hugging Face, we need to add a nice README "dataset card" for it that explains a little about the upstream dataset. I have set up a collection here to keep track of the uploaded sets: https://huggingface.co/collections/UniverseTBD/multimodal-universe-lsdb-hats.

Card list:

• sdss.py
• btsbot.py
• chandra.py
• cfa.py
• csp.py
• des_y3_sne_ia.py
• desi.py
• desi_provabgs.py
• foundation.py
• gaia.py
• [ ] gui.py (not a catalog script)
• gz10.py
• hsc.py
• jwst.py
• legacysurvey.py (very large ~70TB)
• manga.py (very large ~20TB)
• plasticc.py
• ps1_sne_ia.py
• snls.py
• ssl_legacysurvey.py
• [ ] start.py (not a catalog script)
• swift_sne_ia.py
• tess.py
• vipers.py
• yse.py

[Smith42]

Related: #30

[Smith42]

Adding Dataset Cards to HATS collections on HuggingFace

We now have a script (generate_readme.py in the branch #82) that automatically generates HuggingFace Dataset Card READMEs for our HATS collections. Here's how to add a card for each dataset:

1. Generate the README for a dataset (e.g. mmu_gaia_gaia):

   uv run python generate_readme.py https://huggingface.co/datasets/UniverseTBD/mmu_gaia_gaia -o ./readmes/mmu_gaia_gaia/

2. Review the generated README.md — check that the metadata, column descriptions, and code examples look correct.

3. Check the dataset preview. Before opening your PR, verify that the HuggingFace dataset viewer works properly for the dataset. Go to the dataset page (e.g. https://huggingface.co/datasets/UniverseTBD/mmu_gaia_gaia) and check the "Dataset Viewer" tab — make sure the data loads correctly and the columns display as expected. If the preview doesn't work, investigate whether the configs in the YAML frontmatter are pointing to the right data_dir.

4. Add Multimodal Universe attribution. Edit the generated README to add a section linking back to the original Multimodal Universe project. Add the following after the introductory paragraph:

   This dataset is part of the [Multimodal Universe](https://github.com/MultimodalUniverse/MultimodalUniverse),
   a large-scale collection of multimodal astronomical data. For full details, see the paper:
   [The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TBs of Astronomical Scientific Data](https://arxiv.org/abs/2412.02527).

5. Add dataset-specific context. The auto-generated card covers structure and columns but doesn't know about the science behind each dataset. Please review the MMU paper — particularly the section describing the upstream survey/catalogue your dataset comes from — and add relevant information such as:

   • What the original survey is (e.g. Gaia DR3, SDSS-II, DESI, etc.) and a link to its homepage
   • What modality the data covers (images, spectra, light curves, tabular, etc.)
   • Typical science use cases
   • Sample selection criteria or any caveats users should be aware of
   • Citation information for the upstream survey

   This kind of context makes the dataset card much more useful to someone discovering it on HuggingFace.

6. Add a hero figure. Find or create an example visualisation of the data in the dataset — for example, a plot of an example spectrum, a sample image cutout, a light curve, or a sky coverage map. Add the image file to the HuggingFace repo and reference it at the top of the README:

   <div align="center">
   <img src="example_figure.png" width="600">
   </div>

   This gives visitors an immediate visual sense of what the dataset contains. You can generate these from the data itself using the LSDB code examples in the README, or find a suitable figure from the upstream survey's documentation or the MMU paper.

7. Open a PR on the HuggingFace dataset repo to add the README (and hero figure). You can do this from the dataset page — go to e.g. https://huggingface.co/datasets/UniverseTBD/mmu_gaia_gaia, click "Files and versions", then use "Add file" → "Upload files" to upload the files on a new branch and open a PR for review.

8. Repeat for each dataset that needs a card.

You'll need a HuggingFace account to open PRs. Reach out if you have any questions!

[hombit]

@Smith42 Thank you! Feel free to require changes / submit a PR to the hats with any README.md template changes that would help. The current template is not really flexible and supports a single markdown/html blob injection point: dataset description. It may make sense to add customizable headers/footers, etc.

[Smith42]

Some notes on the meeting today that are relevant to this issue:

• We need to put in the PLASTICC README that the RA and DEC are random and not real coordinates
• We can give feedback (and open up PRs or issues on the HATS repo here) on how we could improve the README generation code.
  • For example, do we want to be able to inject some manually written text from the user side into the READMEs?
  • Or would we like to automatically generate skyplots of coverage like the one on lsdb.io be placed in the README?
  • Or would we like to allow users to define assets to be produced inline in the README (like spectra plots etc.)
  • Perhaps we want to alter the README code to read the user's token and automatically produce a PR for the target Hugging Face dataset to add the README?