OAI for Archipelago

[alliomeria]

What?

• Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH
• Develop corresponding documentation

Check another box off the Roadmap!

[DiegoPino]

Nice. Ok, since this is archipelago, tiny little islands where we build stuff everyday using twigs and scrap metal (@alliomeria et Al) so we are going to do things differently that the contributed D8 OAI modules in the wild (here is my plan, lurkers take note!)
Ingredientes:

• First: where does the data come from? From Solr. Most configurable way of getting it in Archipelago? Via a Solr Entity Data View (special thing which we coded, provide and is quite a thing): So a zuper fast data View! (we just need the NODE ids for that)
• Second: What is passed to the Data view? Arguments o course. Each view can have varying arguments and users can be quite creative.
• Third: How do we expose this? A custom Metadata API Endpoint, a special variation of our Metadata Endpoints (yeah, not directly via the View API, that is not our way, we need more control and some cleaner stuff). This custom Metadata API Endpoints use the same idea of the Per Node ones (old good `/do/{uuid}/metadata/{metadata name}/default.{extensionforformat}), but! This kids allow a few new tricks:

You can configure Arguments that are allowed and their format (GET API Builder!):

1. Arguments can affect 4 different things:
   A. Can pass (map) from the API endpoint URL via GET to the View internally (e.g when calling a for a. Set)
   B. Can pass(map) from the API endpoint URL via GET and modify which actual Metadata endpoint is called for each record (IIIF, MODS, etc). Also only ones matching the Primary response type of this API endpoint will be allowed to be set as options. (nice!)
   B. Can pass(map) from the API endpoint URL via GET and feed(injected into) the actual metadata endpoint Twig source/Metadata Display entity (as $context) for cool dynamic rendering and modification and stuff of metadata
2. Allow us also to generate a Wrapper (Also Twig template) Response. All that goes around the actual data.
3. Caching works great on these (because caching will not be applied to the response itself (each Node) but to the actual Node rendering. So add a node and you get immediate response.
4. Permissions are of course enforced and we can dynamically also also roles/assignment as we do for Form Modes (plugin derivers, since this are config entities)

This also solves of course any issues with "Uh, Oh, I need ETDS, can someone code it?" Twig template + add to the config, map the value. Done.

@alliomeria @giancarlobi how do you feel about this?

And other uses (wait for it)

• IIIF Search API (just a wrapper, nice responses)
• OAI-ORE. Set output to JSON-LD, change the argument, done (and using all the same Solr data view as PHM)
• IIIF Collection API Set output to JSON-LD, change the argument, use a different template.

Final crazy idea (which we already discussed with Giancarlo at some moment in the past but I can not find which issue)
https://apisyouwonthate.com/blog/openapi-v31-and-json-schema-2019-09

e.g of an API described via openapi

parameters:
  - in: query
    name: limit
    schema:
      type: integer
      maximum: 50
    examples:       # Multiple examples
      zero:         # Distinct name
        value: 0    # Example value
        summary: A sample limit value  # Optional description
      max: # Distinct name
        value: 50   # Example value
        summary: A sample limit value  # Optional description

My crazy idea is quite simple. The configuration entity will be setup in such a way that the output of one of these will have close shape to the OpenAPI specs to define the arguments (in:query) and what they do.

That way you can even invent your own API. And done.

Good night.

[DiegoPino]

Another note:
This is the custom Views Display (for data) i created that can resolve Entities
https://github.com/esmero/webform_strawberryfield/blob/master/src/Plugin/views/display/EntityReference.php

it is one option. Now that we are natively indexing parent/child relationships as EntityReferences in Solr, we may not even need this any more.

[DiegoPino]

@alliomeria @pcambra @giancarlobi ok, I'm on this now. Always the "frame" around the code is the most expensive part

A few notes for myself where I will start (repeating my issue body really)

1. This implementation will use at least 1 twig template/Metadata displays but normally really 2 as config option.
   a. First template is the Protocol response wrapper. E.g in the case of a OAI will contain the global XML that surrounds individual records
   b. Each record may be rendered individually by a second template. This way we have separation of "expected" API response v/s individual records (a single NODE)

2. The API will have a single (for NOW) view as a setting that will generate/used for generating a listing of resources based on arguments (exposed filters).

3. Configurable Arguments of the API may either map to:

• View Exposed Filter: Count of records, additional arguments we may allow the user to pass. Each argument will be Strictly typed and validated
• A Metadata Display Selection (Mapping Form Element): Certain arguments may/no trigger a change on the Secondary (plural) twig templates, the ones per record.
• Data passed directly to the master: Twig Template and also passed to each record. Also Strictly typed and validated

Not sure I need anything else?

[DiegoPino]

This issue here is growing. I just learned that OAI is a bad bad API. In case a given ID was not found instead of returning a 404 HTTP code it returns a 200! found with a body message. So not much of a real REST API at all.

E.g

dpino@Diegos-Mac-mini metro % curl -iv "http://export.arxiv.org/oai2?verb=GetRecord&identifier=oai:arXiv.org:quant-ph/02131001&metadataPrefix=oai_dc"
*   Trying 128.84.21.203...
* TCP_NODELAY set
* Connected to export.arxiv.org (128.84.21.203) port 80 (#0)
> GET /oai2?verb=GetRecord&identifier=oai:arXiv.org:quant-ph/02131001&metadataPrefix=oai_dc HTTP/1.1
> Host: export.arxiv.org
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Thu, 04 Feb 2021 19:26:48 GMT
Date: Thu, 04 Feb 2021 19:26:48 GMT
< Server: Apache
Server: Apache
< Vary: Accept-Encoding,User-Agent
Vary: Accept-Encoding,User-Agent
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/xml
Content-Type: text/xml

< 
<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
<responseDate>2021-02-04T19:26:49Z</responseDate>
<request>http://export.arxiv.org/oai2</request>
<error code="badArgument">Found illegal duplicate of argument 'amp'.</error>
<error code="badArgument">Verb 'GetRecord', illegal argument 'amp'.</error>
</OAI-PMH>
* Connection #0 to host export.arxiv.org left intact
* Closing connection 0

The specs are terrible http://www.openarchives.org/OAI/openarchivesprotocol.html#ProtocolMessages
Which means to have a flexible API builder in archipelago we may need to allow people to decide on how to respond to exceptions... I was hoping to have to avoid writing body for things that do not check.

[dmer]

friendly request for status update on this - our project has a need to provide an OAI feed to its main harvester/aggregator (Calisphere) by the time we go live (hopefully in September).
We are available and eager to do extensive testing of the OAI feed/configuration!
thanks

[DiegoPino]

@dmer will be ready by next week. Work has started and it is easier than expected. More to come, will also support OAI-ORE