Mention IVOA note, and update example directory. (#602)

Author: delucchi-cmu

docs/citation.rst

@@ -51,6 +51,17 @@ If you used Rubin Data Preview 1 (DP1) with HATS, please also consider citing th
         adsnote       = {Provided by the SAO/NASA Astrophysics Data System}
     }
 
+To cite the HATS format (and not this software package in particular), we suggest the following:
+
+.. code-block:: bash
+
+    @misc{ivoa:NOTE:2025:HATS,
+        author = {Caplar,Neven and DeLucchi, Melissa and  Jurić, Mario and Beebe,Wilson and Branton, Doug and  Campos,Sandro and Jones,  Derek  and  Malanchev,Konstantin and  McGuire,  Sean and  Lynn, Olivia and  Pineau, François-Xavier and  Raen,  Troy},
+        title = {{HATS: A Standard for the Hierarchical Adaptive Tiling Scheme in the Virtual Observatory}},
+        howpublished = {IVOA Note, Version 1.0},
+        year = {2025},
+        note = {Available at \url{https://www.ivoa.net/documents/Notes/HATS/}}
+    }
 
 Acknowledgements
 -------------------------------------------------------------------------------

docs/getting_started.rst

@@ -39,16 +39,16 @@ The latest release version of HATS is available to install with
                 pyenv virtualenv 3.11 hats_env
                 pyenv local hats_env
 
-    We recommend Python versions **>=3.10, <=3.12**.
+    We recommend Python versions **>=3.10, <=3.13**.
 
 HATS can also be installed from source on `GitHub <https://github.com/astronomy-commons/hats>`__.
 
-
 LSDB
 ----
 
-For the most part, we recommend accessing and processing HATS data using the `LSDB package
-<https://github.com/astronomy-commons/lsdb>`__ framework. LSDB provides a variety of utility
-functions as well as a lazy, distributed execution framework using Dask.
+For the most part, we recommend accessing and processing HATS data using the 
+`LSDB framework package <https://github.com/astronomy-commons/lsdb>`__.
+LSDB provides a variety of utility functions as well as a lazy, 
+distributed execution framework using Dask.
 
-For detail on LSDB, see the `readthedocs site <https://docs.lsdb.io/en/stable/>`__.
+For details on LSDB, see the `readthedocs site <https://docs.lsdb.io/en/stable/>`__.

docs/guide/directory_scheme.rst

@@ -1,6 +1,11 @@
 HATS Directory Scheme
 ===============================================================================
 
+You can read in more detail about the parts of the HATS directory structure
+in the `IVOA Note <https://www.ivoa.net/documents/Notes/HATS/>`__.
+This page provides a summary of what you can expect inside a HATS-structured
+catalog.
+
 Partitioning Scheme
 -------------------------------------------------------------------------------
 
@@ -33,22 +38,22 @@ structure:
 
 .. code-block:: 
     :class: no-copybutton
-    
-    __ /path/to/catalogs/<catalog_name>/
-       |__ partition_info.csv
-       |__ properties
-       |__ dataset/
-           |__ _common_metadata
-           |__ _metadata
-           |__ Norder=1/
-           |   |__ Dir=0/
-           |       |__ Npix=0.parquet
-           |       |__ Npix=1.parquet
-           |__ Norder=J/
-               |__ Dir=10000/
-                   |__ Npix=K.parquet
-                   |__ Npix=M.parquet
 
+      /path/to/catalogs/<catalog_name>/
+      ├── dataset
+      │   ├── _common_metadata
+      │   ├── _metadata
+      │   ├── Norder=0
+      │   │   └── Dir=0
+      │   │       ├── Npix=0.parquet
+      │   │       └── Npix=1.parquet
+      │   └── Norder=J
+      │       └── Dir=10000
+      │           ├── Npix=K.parquet
+      │           └── Npix=M.parquet
+      ├── hats.properties
+      ├── partition_info.csv
+      └── skymap.fits
 
 As you can notice, ``dataset/`` has the following heirarchy:
 
@@ -60,3 +65,38 @@ As you can notice, ``dataset/`` has the following heirarchy:
    Note: instead of being a single Parquet file, this can be a directory containing
    one or more Parquet files, representing a single data partition, i.e., they should
    be read together as a single data unit.
+
+Collections
+-------------------------------------------------------------------------------
+
+HATS also makes use of supplemental tables for catalogs, and these are grouped into
+catalog Collections. 
+
+.. code-block:: 
+    :class: no-copybutton
+
+      /path/to/catalogs/<catalog_name>/
+      ├── collection.properties
+      ├── primary_catalog
+      │   ├── dataset
+      |   |   └── ...
+      │   ├── hats.properties
+      │   ├── partition_info.csv
+      │   └── skymap.fits
+      ├── id_index
+      │   ├── dataset
+      |   |   └── ...
+      │   └── hats.properties
+      └── margin_10arcs
+         ├── dataset
+         |   └── ...
+         ├── hats.properties
+         └── partition_info.csv
+
+The ``primary_catalog`` directory will be the same format as the catalog described above.
+Here we have additional supplemental tables:
+
+* ``id_index`` - a secondary index to enable finding rows by non-spatial queries
+* ``margin_10arcs`` - a cache of rows that are within a limited angular threshold
+  around the primary catalog data partition, useful for cross-matching to catalogs
+  with slightly different astrometry.

docs/index.rst

@@ -5,6 +5,10 @@ HATS is a directory structure and metadata for spatially arranging large catalog
 This was originally motivated by a desire to perform spatial cross-matching between surveys 
 at large scale, but is applicable to a range of spatial analysis and algorithms.
 
+You can read in more detail about the parts of the HATS directory structure
+in the `IVOA Note <https://www.ivoa.net/documents/Notes/HATS/>`__. This library is simply
+one implementation of interacting with HATS-formatted datasets.
+
 We use healpix (`Hierarchical Equal Area isoLatitude Pixelization <https://healpix.jpl.nasa.gov/>`__)
 for the spherical pixelation, and adaptively size the partitions based on the number of objects.
 Each partition will have roughly the same number of objects, instead of dividing into equal area. 

src/hats/pixel_math/partition_stats.py

@@ -282,9 +282,10 @@ def generate_incremental_alignment(
         one-dimensional numpy array of integer 3-tuples, where the value at each index corresponds
         to the destination pixel at order less than or equal to the mapping order.
         The tuple contains three integers:
-            - order of the destination pixel
-            - pixel number *at the above order*
-            - the number of objects in the pixel
+
+        - order of the destination pixel
+        - pixel number *at the above order*
+        - the number of objects in the pixel
     """
     _validate_alignment_arguments(histogram, highest_order, lowest_order, threshold)