[TRACKER]: Upload converted datasets to Hugging Face

[Smith42]

Once we have the data modality checked off in issue #13, we need to compile to HATS and upload the dataset to Hugging Face. I have set up a collection here to keep track of the uploaded sets: https://huggingface.co/collections/UniverseTBD/multimodal-universe-lsdb-hats. Please comment below if you have uploaded a set and I will add it to the collection.

Upload list:

• sdss.py
• btsbot.py
• chandra.py (https://users.flatironinstitute.org/~polymathic/data/MultimodalUniverse/v1/chandra/chandra.py forbidden)
• cfa.py (has bug to be fixed that appears when compiling: Illegal slicing argument for scalar dataspace)
• csp.py (has bug to be fixed that appears when compiling: Illegal slicing argument for scalar dataspace)
• des_y3_sne_ia.py
• desi.py
• desi_provabgs.py
• foundation.py
• gaia.py
• [ ] gui.py (not a catalog script)
• gz10.py
• hsc.py
• jwst.py
• jwst.py GRIZLI (grizli catalogues have bug "KeyError('image_flux')" Traceback: ' File "mmu-hdf-to-hats/.venv/lib/python3.13/site-packages/hats_import/catalog/map_reduce.py", line 306, in split_pixels")
• legacysurvey.py (very large ~70TB)
• manga.py (very large ~20TB)
• plasticc.py
• ps1_sne_ia.py
• snls.py
• ssl_legacysurvey.py
• [ ] start.py (not a catalog script)
• swift_sne_ia.py
• tess.py
• vipers.py
• yse.py

EDIT: added chandra, crossed out gui and start, which are both related to a streamlit server and no catalogs

[Smith42]

I am attempting to compile Legacy Survey, but am running into issues in the "Reducing" stage. Possibly due to the total number of files (21177), or due to a lack of memory? Perhaps we need to have a think about how we compile very large datasets, as Legacy Survey is ~70TB.

Edit: time to completion is >200 days. We will need to have a think about how to compile.

[Smith42]

TESS fails with the error:

2026-01-06 19:06:49,387 - distributed.worker - ERROR - Compute Failed
Key: map_to_pixels-996a7d959068ce2389d13452a8a80233
State: executing Task: <Task 'map_to_pixels-996a7d959068ce2389d13452a8a80233' map_to_pixels(, ...)>
Exception: 'KeyError("Unable to synchronously open object (object 'ra' doesn't exist)")'

[Smith42]

gaia fails due to some HDF5 files within the MMU being corrupted. Fixed by ignoring healpix 1216 in compilation.

[CloseChoice]

I am attempting to compile Legacy Survey, but am running into issues in the "Reducing" stage. Possibly due to the total number of files (21177), or due to a lack of memory? Perhaps we need to have a think about how we compile very large datasets, as Legacy Survey is ~70TB.

Edit: time to completion is >200 days. We will need to have a think about how to compile.

Is it possible to upload only a specific healpix of legacy? I would assume the problem here is that hats doesn't understand that the data is already partitioned in a similar way as required by hats. So maybe uploading in chunks would work and speed up the process significantly (e.g. first x healpixels, then healpixels adjacent to that, etc.)

[Smith42]

That is a great idea @CloseChoice, I was worried that this would mean that some galaxies go missing at the healpix edges however?

I will bring it up on the Discord...

[CloseChoice]

TESS fails with the error:

2026-01-06 19:06:49,387 - distributed.worker - ERROR - Compute Failed Key: map_to_pixels-996a7d959068ce2389d13452a8a80233 State: executing Task: <Task 'map_to_pixels-996a7d959068ce2389d13452a8a80233' map_to_pixels(, ...)> Exception: 'KeyError("Unable to synchronously open object (object 'ra' doesn't exist)")'

Hi Mike,
I assume you fixed this since tess is uploaded already. Anything else I can look into atm?

[CloseChoice]

@Smith42 Hey, I had a look at the python transformer scripts and updated the list, since we have 2 files in there that have nothing to do with catalogs, they are just there for visualization (start.py and gui.py, I could have guessed from the names ;). Also updated #13, off-topic but this leaves just 1 (EDIT: all done!) transform script remaining!!!

EDIT: the chandra script cannot be downloaded from the flatiron server (https://users.flatironinstitute.org/~polymathic/data/MultimodalUniverse/v1/chandra/chandra.py). You mentioned that we could get them from the original repo but I am a bit lost where this is.

EDIT2: found the chandra.py in an old commit of this repo.

[Smith42]

Hey @CloseChoice so nice that we are all done!!!! I will upload the smaller datasets from my end 😃. We need permission from HF to upload Legacy and MANGA, which we are working on

I see you found it but for future reference (for others) the chandra.py file is available here https://github.com/MultimodalUniverse/MultimodalUniverse/tree/main/scripts/chandra

And the other scripts are in https://github.com/MultimodalUniverse/MultimodalUniverse/tree/main/scripts

[Smith42]

NOTES: cfa datasets have a bug to be fixed: Illegal slicing argument for scalar dataspace

[CloseChoice]

NOTES: cfa datasets have a bug to be fixed: Illegal slicing argument for scalar dataspace

could you pls provide a script to reproduce? I assume you run the upload script but a direct command would help, then I'll take a look soon

[Smith42]

Yep here are the scripts for the bugged datasets:

uv run ./main.py --transformer=jwst --input=MultimodalUniverse/v1/jwst/ceers --output=./hats --name=mmu_jwst_ceers --max-rows=8192

Error:
Byte-swapped arrays not supported 2026-02-11 19:58:21,514 - distributed.worker - ERROR - Compute Failed Key: map_to_pixels-a5406d775fffb85ce79580ae3de9ab88 State: executing Task: <Task 'map_to_pixels-a5406d775fffb85ce79580ae3de9ab88' map_to_pixels(, ...)> Exception: "ArrowNotImplementedError('Byte-swapped arrays not supported')"

uv run ./main.py --transformer=cfa --input=MultimodalUniverse/v1/cfa/cfa3 --output=./hats --name=mmu_cfa_cfa3 --max-rows=8192

and

uv run ./main.py --transformer=csp --input=MultimodalUniverse/v1/csp/csp --output=./hats --name=mmu_csp_csp --max-rows=8192

both result in

Illegal slicing argument for scalar dataspace

[CloseChoice]

@Smith42 I have a fix for csp in here: #74. As it states, cfa is still WIP and a couple of questions are not yet answered for me, so this is more of a FYI that I am working on it.

[CloseChoice]

@Smith42 justed merge #74 which fixed csp & cfa. I ran the code locally and that worked. Sorry, I didn't really spend time how to upload those. If you don't find the time to upload, I'll take a deeper look how to do that, don't mean to offload work to you ;)