[TRACKER]: Verify transformer_<catalog>.py output matches <catalog>.py output

[CloseChoice]

This is a tracker issue for all catalog.py files.
Each catalog.py file can be picked up individually and should result in exactly one PR. The transform_scripts/transform_<catalog>_to_parquet.py have been written by claude, so they need to be checked thoroughly since they can contain mistakes.

Here is a reference PR: #53

To start, do the following:

• choose a non-checked script (see list below) from https://github.com/UniverseTBD/mmu-hdf-to-hats/tree/main/catalog_download_scripts
• then check the corresponding catalog on https://users.flatironinstitute.org/~polymathic/data/MultimodalUniverse/v1 and find a small healpix (<100MB is ideal)
• write a download script, e.g. verification/download_sdss.sh using the healpix you found
• write a process__using_datasets.py, execute it using the command uv run --with-requirements=verification/requirements.in python verification/process_<catalog>_using_datasets.py (this is important, since it will run with datasets==3.6 which is the last version to support custom scripts)
• write a catalog_functions/_transformer.py
• write a transform_scripts/transform__to_parquet.py and run it (simply using python transform_scripts/transform_<catalog>_to_parquet.py is fine here)
• run python verification/compare.py <path1> <path2> where the paths are the output paths of process_<catalog>_using_datasets.py and transform_scripts/transform_<catalog>_to_parquet.py
• add the catalog to the CI workflow here
• add the corresponding files in verify.py, see here

Problems that can arise:

• in desi a negation operator for a boolean column was missing
• float conversion can be problematic due to different float types
• the object id matching when adding the coordinates can be off, since object_ids are differently formatted across the catalogs

Tracker list:

• sdss.py
• btsbot.py
• cfa.py
• csp.py
• chandra.py (https://users.flatironinstitute.org/~polymathic/data/MultimodalUniverse/v1/chandra/chandra.py forbidden but can be retrieved from commit 2dcff3d of this repo)
• des_y3_sne_ia.py
• desi.py
• desi_provabgs.py
• foundation.py
• gaia.py
• [ ] gui.py (not a catalog script)
• gz10.py
• hsc.py
• jwst.py
• legacysurvey.py
• manga.py
• plasticc.py
• ps1_sne_ia.py
• snls.py
• ssl_legacysurvey.py
• [ ] start.py (not a catalog script)
• swift_sne_ia.py
• tess.py
• vipers.py
• yse.py

[CloseChoice]

working on yse

EDIT: skipping for now due to this error:

  File "/home/tpitters/programming/toy-mmu/mmu-to-hats/verification/process_yse_using_datasets.py", line 12, in <module>
    yse_catalog = get_catalog(yse)
  File "/home/tpitters/.cache/uv/archive-v0/tAokYlpcyMSj8hNTmXY7i/lib/python3.13/site-packages/mmu/utils.py", line 44, in get_catalog
    catalogs.append(_file_to_catalog(filename, keys=keys))
                    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/tpitters/.cache/uv/archive-v0/tAokYlpcyMSj8hNTmXY7i/lib/python3.13/site-packages/mmu/utils.py", line 16, in _file_to_catalog
    return Table({k: data[k] for k in keys})
  File "/home/tpitters/.cache/uv/archive-v0/tAokYlpcyMSj8hNTmXY7i/lib/python3.13/site-packages/astropy/table/table.py", line 906, in __init__
    init_func(data, names, dtype, n_cols, copy)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tpitters/.cache/uv/archive-v0/tAokYlpcyMSj8hNTmXY7i/lib/python3.13/site-packages/astropy/table/table.py", line 1457, in _init_from_dict
    self._init_from_list(data_list, names, dtype, n_cols, copy)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tpitters/.cache/uv/archive-v0/tAokYlpcyMSj8hNTmXY7i/lib/python3.13/site-packages/astropy/table/table.py", line 1279, in _init_from_list
    self._init_from_cols(cols)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/tpitters/.cache/uv/archive-v0/tAokYlpcyMSj8hNTmXY7i/lib/python3.13/site-packages/astropy/table/table.py", line 1501, in _init_from_cols
    lengths = {len(col) for col in cols}
               ~~~^^^^^
TypeError: len() of unsized object

Taking on vipers now

[CloseChoice]

working on tess

[CloseChoice]

For swift_sne_ia I only find the following columns:

['DEC_HOST', 'FILTERS', 'IAUC', 'MAG', 'MAGERR', 'MWEBV', 'NOBS', 'PHOTOMETRY_VERSION', 'RA_HOST', 'REDSHIFT_HELIO', 'SEARCH_PEAKMJD', 'SNID', 'SURVEY', 'VPEC', 'bands', 'dec', 'flux', 'flux_err', 'healpix', 'host_log_mass', 'obj_type', 'object_id', 'ra', 'redshift', 'time']

This doesn't match the swift_sne_ia.py file where lightcurve should be there.