Sunset schemas

[WilliamJamieson]

We should consider the conditions under which we fully sunset schemas, that is drop support for them completely. Maybe even remove them from the published RAD package?

Ideally, I would like us to drop support for all of the schemas in datamodels-1.4.0 and below. These all use the nested tag structure and involve non-object tags like tagged lists and tagged scalars. Supporting these introduces a lot of complexity in RDM to simply support rather old files.

Note that the TVAC and FPS schemas are an annoying exception to this as those are already archived. In those special cases we can add special handling to side step issues; however, doing that in general for all the schemas will be more complex than keeping the current system in place. The best case for TVAC and FPS would be for us to have a "bumped" version of them which simply removes the internal tag structure that would be introduced if that data is ever reprocessed. In any case we may wish to define "internal-tag-less" versions of those schemas so that if users wish to modify existing TVAC and FPS data files then they can easily be saved.

@schlafly, @PaulHuwe, and @tddesjardins what are your thoughts?

[braingram]

Reference files on CRDS should also be considered as part of this as some (not sure how many) of those files contain old tag versions. For example the skycells reference file:
https://roman-crds.stsci.edu/browse/roman_wfi_skycells_0001.asdf
contains:

roman: !<asdf://stsci.edu/datamodels/roman/tags/reference_files/skycells-1.0.0>
...
    origin: !<asdf://stsci.edu/datamodels/roman/tags/origin-1.0.0> STSCI
    pedigree: GROUND
    pixel_scale: 0.055
    reftype: SKYCELLS
    skycell_border_pixels: 100
    telescope: !<asdf://stsci.edu/datamodels/roman/tags/telescope-1.0.0> ROMAN

[schlafly]

I think conceptually it would be fine to drop support for sufficiently old reference files and data files, though I would want them to be quite old. I find it very frustrating when my existing files break. But I agree that we don't want to keep a lot of cruft around forever.

As you say, we do need to at least support the TVAC / FPS data ~indefinitely. I don't think I would consider dropping support for our old test files until ~1 year after launch when people have lost most of their appetite for old simulated files.

[PaulHuwe]

Maybe we should save removing them for major version updates? Seems a pretty substancial change to tuck in a lesser version.

[tddesjardins]

Just to be clear, we're saying this for pre-launch, correct? Once we're in operations, I do believe we will need to maintain things ~indefinitely. For reference files, it's also complicated as we need to update reference files to work with a given pipeline version, but CRDS does have a history of older files in it. That said, we cleaned out CRDS recently to have things current as of something like Build 18 or so, and RTB is working to make the final day-zero (into commissioning) reference files to replace things on CRDS, but older contexts would retain the existing files.

I'm also concerned about communicating this change with anyone who has made their own simulations with older versions of SOC code. So I think there will be some amount of socializing this sunset policy with the Software WG via Harry and Eddie.

@rgcosentino can you weigh in here about any concerns you have?

[schlafly]

Yes, William's proposal would only be for pre datamodels-1.4.0 schemas, ~8 months ago.

[rgcosentino]

Sounds like an excellent topic for discussion at an upcoming science quarterly and software working group. Per @tddesjardins concerns, how much would such a change impact current community development and support for core community survey software development and analysis tools.