SSC ROMAN-3776 Update GDPS data models for G1DP

[gduggan-caltech]

Resolves SSC's ROMAN-3776

This PR addresses the feedback from SSC GDPS G1DP on the wfi_spec_catalog_level_4.yaml and wfi_spec_data_quality_level_4.yaml data models.

Details:

• Rename id_catalog with catalog_id to match SOC naming convention of having id appear at the end of the keyname
• Rename PROGNUM with program to match rad field.
• Rename PROGRAM_COMPLETE with program_complete to have all keys be lower case.
• Remove ID_OBS, FILESET_ID, SEGMENT, and VISNUM from observation file, because the catalog is not tied to a specific OBS_ID. Those keys have been moved to g2dp_common for G2DP to provide feedback on.

Tasks

Not sure which of the following tasks need to be performed:

• Update or add relevant rad tests.
• Update relevant docstrings and / or docs/ page.
• Does this PR change any schema files?
  • Schema changes were discussed at RAD Review Board meeting.
• Does this PR change any API used downstream? (If not, label with no-changelog-entry-needed.)
  • Write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types).
  • Start a romancal regression test (https://github.com/spacetelescope/RegressionTests/actions/workflows/romancal.yml) with this branch installed ("git+https://github.com/<fork>/rad@<branch>").
  • Update relevant roman_datamodels utilities and tests.

News fragment change types:

• changes/<PR#>.feature.rst: new feature
• changes/<PR#>.bugfix.rst: fixes an issue
• changes/<PR#>.doc.rst: documentation change
• changes/<PR#>.removal.rst: deprecation or removal of public API
• changes/<PR#>.misc.rst: infrastructure or miscellaneous change

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.48%. Comparing base (93a8654) to head (bd8de90).
⚠️ Report is 156 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #701   +/-   ##
=======================================
  Coverage   97.48%   97.48%           
=======================================
  Files           8        8           
  Lines         716      716           
=======================================
  Hits          698      698           
  Misses         18       18           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[braingram]

Thanks for the PR. Would you add a changelog entry?

The romancal test failure is unrelated.

[braingram]

Perhaps @cjarnold can review this one. Do the changes here align with the other SSC schema updates?

[gduggan-caltech]

This comment can be part of a different PR if you prefer. If this is a redundant question, please point me to where I should be looking.

I'm trying to test that this data model matches what I expect.

Here's an example json for both the wfi_spec_catalog_level_4-1.0.0 and wfi_spec_data_quality_level_4-1.0.0 models titled
gdps_g1dp_level_4_metadata_example.json:

{
    "program": "00002",
    "program_complete": "False",               
    "file_date": "2023-08-05T12:53:27.658",
    "origin": "IPAC/SSC",
    "filename": "wfi_spec_catalog_prism_00002_0001_f0001.parquet",
    "optical_element": "PRISM",
    "catalog_id": "0001"
}

And then I ran this little test script:

from importlib.resources import files
import json

import asdf
import asdf_standard


def test_wfi_spec_catalog_level_4():
    # add ssc resources
    resources = asdf.resource.DirectoryResourceMapping(
        files("rad") / "resources" / "schemas" / "SSC",
        "asdf://stsci.edu/datamodels/roman/schemas/SSC/",
        recursive=True,
    )

    asdf.get_config().add_resource_mapping(resources)

    asdf.AsdfFile(
        {"meta": json.load(open("gdps_g1dp_level_4_metadata_example.json"))},
        custom_schema="asdf://stsci.edu/datamodels/roman/schemas/SSC/GDPS/wfi_spec_catalog_level_4-1.0.0",
    ).validate()


def test_wfi_spec_data_quality_level_4():
    # add ssc resources
    resources = asdf.resource.DirectoryResourceMapping(
        files("rad") / "resources" / "schemas" / "SSC",
        "asdf://stsci.edu/datamodels/roman/schemas/SSC/",
        recursive=True,
    )

    asdf.get_config().add_resource_mapping(resources)

    asdf.AsdfFile(
        {"meta": json.load(open("gdps_g1dp_level_4_metadata_example.json"))},
        custom_schema="asdf://stsci.edu/datamodels/roman/schemas/SSC/GDPS/wfi_spec_data_quality_level_4-1.0.0",
    ).validate()

1. Can we add this unit test somewhere?
2. It's upset about the file_date. What do we need to do to fix this? See error message below.

asdf._jsonschema.exceptions.ValidationError: mismatched tags, wanted 'tag:stsci.edu:asdf/time/time-1.*', got 'tag:yaml.org,2002:str'

Failed validating 'tag' in schema['properties']['meta']['allOf'][0]['properties']['file_date']:
    {'archive_catalog': {'datatype': 'datetime2',
                         'destination': ['CGIExposure.filedate',
                                         'CGIMosaic.filedate',
                                         'CGIAncillary.filedate']},
     'description': 'The file creation date as taken from the '
                    "'fileCreationTimestamp' property of the\n"
                    'data availability notification '
                    '(https://github.com/spacetelescope/roman-soc-ssc-common/blob/main/schemas/data_availability.json#L76)\n'
                    "SSC does not need to include this in the 'metadata' "
                    'object of the data availability notification.\n',
     'tag': 'tag:stsci.edu:asdf/time/time-1.*',
     'title': 'File Creation Date (UTC)'}

On instance['meta']['file_date']:
    '2023-08-05T12:53:27.658'

[ketozhang]

Related to #699 (replaces tag with $ref to fix validation for JSON serializables)

[braingram]

Yeah this is related to #699 as @ketozhang mentioned. It appears that the manifest does not contain tagged times (so the tag validator will always fail). Instead this manifest contains the string "2023-08-05T12:53:27.658" which appears to be an iso formatted time.

Can we add this unit test somewhere?

What did you have in mind for this test? Is the code that you use to generate manifest public so that we could generate a manifest and test it with the schema?

[cjarnold]

Perhaps @cjarnold can review this one. Do the changes here align with the other SSC schema updates?

Added a couple comments- thank you for the PR @gduggan-caltech. I have a GDPS PR https://github.com/spacetelescope/rad/pull/699/files which I suspect I should limit to just the changes I have made so far there and get that merged into main, along with your PR so we have a clean start for G2DP.

[ketozhang]

What did you have in mind for this test? Is the code that you use to generate manifest public so that we could generate a manifest and test it with the schema?

Providing the generation code is unlikely needing to publish our internal data models.

We can provide and maintain a set of example static manifest for each model. We have been providing examples for the past few months now, and having these be integrated into tests serves as a good contract testing between the SSC science pipelines & RAD (w/o having to involve the exchange system).

[gduggan-caltech]

Let me know if there's anything your waiting on me for! It's fine with me if these changes are merged as-is and testing scripts are dealt with in a future PR.

[PaulHuwe]

All SSC tickets are waiting for review by @ejoliet.

[ejoliet]

Changes/updates look fine to me.

[PaulHuwe]

Needs a review from @jbrookens or @kdupriestsci for the archive.

[gduggan-caltech]

Changes look good to me. Here's the current wfi_spec_catalog_level_4 and wfi_spec_data_quality_level_4 example:

{
    "program": 2,
    "program_complete": false,               
    "file_date": "2023-08-05T12:53:27.658",
    "origin": "IPAC/SSC",
    "filename": "wfi_spec_catalog_prism_00002_0001_f0001.parquet",
    "optical_element": "PRISM",
    "catalog_id": "0001"
}

[gduggan-caltech]

Can we make progress on this PR? It would be nice to have this merged before G2DP starts making their own PR. However, if that's not feasible, just let us know.

[jbrookens]

Archive changes aren't in a stable enough place where approval is appropriate. We're still trying to figure out what the file types are and where the metadata will live in the archive.

[ejoliet]

Archive changes aren't in a stable enough place where approval is appropriate. We're still trying to figure out what the file types are and where the metadata will live in the archive.

Thanks. Do you have a timeline for this approval to occur? Thanks again.

[PaulHuwe]

LGTM

[PaulHuwe]

Approved and merged.