Mega-issue: Vendorization of the dandi-schema and other components

[yarikoptic]

This issue provides an upper-level view of the plan and progress to make metadata models in the DANDI ecosystem vendor-configurable organized by release stages.

Closed stages: 1, 2

Release Stage 1:

Changes in this stage even if merged & released do not effect existing setup/functionality.

PRs could be merged independently.

• dandi-archive
  • Local runtime schema serialization endpoints dandi-archive#2386
• dandi-infrastructure: Set environment vars to set vendor-specific dandi-schema used by the DANDI instance dandi-infrastructure#224
• linc-archive-infrastructure: Set environment vars to set vendor-specific dandi-schema used by the DANDI instance lincbrain/linc-archive-infrastructure#24 (status: to be merged)
• ember-dandi-infrastructure: Set environment vars to set vendor-specific dandi-schema used by the DANDI instance aplbrain/dandi-infrastructure#47

Release Stage 2:

Changes, if merged and released, would affect functionality and provide vendorization for the archive instances.

Components must have coordinated release(s).

Release Stage 2a:

• dandi-schema:
  • Vendor-Configurable Metadata Models #294

Release Stage 2b:

The following must be released at the same time.

• dandi-archive
  • Add vendorization support dandi-archive#2584 (status: to be released)
    • Make sure dandischema is pin to the latest version beyond "0.12.0".
  • upgrade to the released dandi-schema (this is a step in operation)
  • release!
• dandi-cli
  • Set dandischema dependency to 0.12.0 and beyond dandi-cli#1744
  • Review operation and codebase for further hardcoded DANDI identifiers dandi-cli#1752
    • Remove hardcoded vendor specific information dandi-cli#1760
  • release!

Stage X: good to have etc

Could be done now or later conditional on available dandi-schema release

• dandi-schema
  • add otherIdentifiers to Dandiset model #358
• dandi-cli
  • Minor concern about dandi-cli with a version of dandi-schema before the vendorization effort when connecting to a DANDI instance other than DANDI Archive, such as EMBER-DANDI, with a version of dandi-schema after the vendorization effort may generate errors.
  • Add vendor-specific behavior to dandi validate and `dandi upload, validation - add support for vendorization record from the instance server dandi-cli#1625
  • Make the dandi service-scripts publish-dandiset-version-doi command DANDI instance specific, incorporating vendor information of a particular DANDI instance dandi-cli#1704 (status: WiP)
    • Vendorize dandi-cli per config in server info dandi-cli#1661 (status: to be revived)
• dandi-archive
  • Along with manifests, dump (vendorized) schema + context to S3 dandi-archive#2577

[candleindark]

@yarikoptic Two issues we need to attend to:

• 1. Is there any plan for continue use of https://github.com/dandi/schema after the vendor-configuration effort? In Local runtime schema serialization endpoints dandi-archive#2386, we have replaced the need of https://github.com/dandi/schema for the distributing dandiset.json. However, dandi-archive still needs the context.json to be hosted at https://github.com/dandi/schema unless additional changes are provided.

  https://github.com/dandi/dandi-archive/blob/31fac2f78c75b9ee5708fabaaee02d417300bd4a/dandiapi/api/models/asset.py#L262-L265

  https://github.com/dandi/dandi-archive/blob/31fac2f78c75b9ee5708fabaaee02d417300bd4a/dandiapi/api/models/version.py#L234-L237

• 2. https://github.com/dandi/dandi-archive/blob/master/web/src/types/schema.ts is generated by running https://github.com/dandi/dandi-archive/blob/master/web/src/migrate.ts on a dandiset.json. We need to check if the generated schema.tss for different vendor-specific dandiset.jsons are the same. If they are different we may have a problem.

  • A possible way to do this to run yarn migrate on different vendor-specific dandiset.json

[yarikoptic]

1. I think solution would be at the level of dandi-archive dumps Along with manifests, dump (vendorized) schema + context to S3 dandi-archive#2577
2. If that schema.ts is used only for development, since it already includes instance-configuration specific metadata (such as licenses https://github.com/dandi/dandi-archive/blob/master/web/src/types/schema.ts#L205C1-L205C61 ) - such file then should be specific to the instance if someone would be developing using it. @mvandenburgh @jjnesbitt -- is that file used only for development or somehow also instructs front end at run time? (how is it used BTW -- can't see from quick search)