Rename referenceObject / $ref property

[MikeRalphson]

From the README and https://github.com/OAI/moonwalk/blob/bd4e2ac3edef2226ae54c22c88d4d90f3de1079f/examples/aggregatedapi.yaml#L6

As the OpenAPI reference object is now independent to the JSON Schema $ref, we are free to support OpenAPI Reference arrays as well as reference objects. This will allow reusing a set of requests or a set of responses.

Now might be the time to consider properly divorcing our referenceObject from JSON Schema's $ref keyword:

• What looks like three properties with the same name but different behaviours is confusing
• Maintaining referenceObject as a strict subset of a JSON Schema schema is problematic enough, if we go back to subset/superset/sideset we push more pain on to our consumers
• Naive bundling tools currently fail to distinguish between $refs in schemaObjects, referenceObjects and pathItemObjects leading to incorrect behaviour
• Forms such as repeated $ref keywords or allowing allOf in the referenceObject have been proposed before for composing the paths object
• We could take the opportunity to remove the special-casing of the $ref property/field in the pathItem object which has more undefined than defined behaviours
• $ref currently supports URIs, but we lack $id/$anchor properties in our referenceObject so we limit JSON Schema compatibility. Is there actually a case for using URIs to compose OAS documents (rather than relative references or absolute URLs)? If we changed this, we would have to specify that $ref URLs were relative to the referring document, not the servers.url (that they are design-time not run-time URLs)
• Our referenceObject only allows processing of description and summary properties, it might be helpful to formally document which properties/fields in OAS are annotations/metadata and which are "concrete" data, and allow all annotations
• We could take the opportunity to limit internal JSON Pointers to the components object, indicating that we are importing from a library of components which just happens to be held within the same OAS document
• We could clarify if exampleObject externalValue UR[I|L]s should always or optionally be fetched to bundle a complete OAS document

I would propose a use, @use, import or @import keyword in a renamed importObject, which for simplicity could always be an array. If we allow variant object and array forms, we should follow the same pattern for externalDocs for consistency. The object form would always behave in the same manner as a 1-element array.

[whitlockjc]

I do not have an opinion on the proposal of how we move on from $ref but I definitely agree that we should. Should that change, I will comment more on the implementation proposed above.

[handrews]

I'd argue that the current referencing situation is even more complicated that the three different "$ref"s. I'll have a lot more to say on this in the next day or so.

There's something worth doing here, but I might turn it sideways a bit. I think the interesting distinctions include semantic vs structural referencing, and static vs runtime resolution. There are several distinct needs, and the existing referencing mechanisms (including but not limited to the "$ref"s) don't map cleanly to those needs.

@MikeRalphson I think this question is key:

Is there actually a case for using URIs to compose OAS documents (rather than relative references or absolute URLs)?

As I said, I'll have more ideas on this soon. For now, I'll just comment on a few minor points:

If we changed this, we would have to specify that $ref URLs were relative to the referring document, not the servers.url (that they are design-time not run-time URLs).

This is already true for Reference Objects in OAS 3.1 per §4.6 Relative References in URIs (contrast with §4.7 Relative References in URLs, note "URLs" vs "URIs" which was perhaps not the most clear way to explain this distinction in retrospect, although it is accurate.)

We could clarify if exampleObject externalValue UR[I|L]s should always or optionally be fetched to bundle a complete OAS document

For anything that might run in a security-conscious environment, it's important to not mandate fetching while processing the file. I/O operations (especially network ones) might be significantly restricted, or required to be performed at a predictable point in the code rather than while processing a file that may or may not be trusted. I once told a sr. security engineer about people wanting to fetch HTTP URLs on demand while validating API payloads with JSON Schema and she made a very entertaining face of horror 😱

Bundling a document might be of less concern than payload validation, but it's hard to know how people will use things. It's a lot more flexible to treat all URIs as identifiers and allow for different approaches to resolving them. Resolution might be to treat them as URLs and dereference them, or to map URLs to other loading mechanisms, or to require everything to be pre-populated and resolved from an in-memory cache. Or a mix of all three.

[darrelmiller]

Yes! This is definitely the right time to think about make a change here. I believe there is a significant opportunity for both simplification and increasing the semantics we represent.

We could take the opportunity to limit internal JSON Pointers to the components object, indicating that we are importing from a library of components which just happens to be held within the same OAS document

If we do this, we can a) avoid the need for the ugly syntax with URLs b) prevent people from referencing incompatible types c) make a clean separation between importing concepts from external documents vs doing internal referencing.

The more I think about these things, the more questions I have. I'm just going to try and list the confusion in my head at the moment:

• Can we make importing of external files a top level concept? Do we really need the ability to import just a fragment of another file? It isn't really viable for tooling to only read a fragment of a file, so what is the value of importing fragments? Now that JSON Schemas can be referenced by Id, we could import JSON Schema files at the top level and then just reference them by Id.
• I'm not really sure I understand the distinction being by runtime vs design time references.
• When we talk about structural vs semantic reuse, does that correlate to inheritance vs inclusion (like is done with the spread operator) ?
• Is this the right time to talk about "generics" support? How do we implement OkResponse<TodoItem> ?
• In client code generation, a distinction is often made between schemas that are inlined in responses and those that are direct component references. Should this become a first class distinction?

[handrews]

@darrelmiller regarding the need for ugly URL syntax (or lack thereof), I think there are really two cases: open world linking, which is particularly relevant when connecting files that are independently managed (think schema libraries), and linking within a known data set. We need URIs for the open world case, or else we'll end up re-inventing them. But we don't necessarily need the for the internal case, and having a more intentional separation might be benefiicial.

Note that I said URIs and not URLs, as it's extremely important to separate identification and location (for example, security people get twitchy if evaluating an API description causes random network traffic at runtime in high-security environments. At one company I worked for all API descriptions and schemas had to be packaged on the appliance as they ran in an air-gapped environment.

The interesting thing about internal references is that there are usually semantic constraints on them, and taking a different approach to the two sorts of references could facilitate safeguards for those references. As opposed to importing, where you're making some new thing available for use, and as long as it parses into recognizable things, that's valid.

Some answers to your questions off the top of my head (I may change my mind on things later):

• I think there's a distinction between importing a file (and making its contents available) and referencing something within the file. I don't see the need to import partial files, but referencing locations within a file is needed. This gets to the open world (import file) vs internal (reference something within the set of loaded documents) distinction. URIs combine these by using fragments, but we could also look to how programming languages allow importing whole modules vs elements within modules for other approaches.
• design time is $ref, runtime is $dynamicRef - more on this a few questions down
• I think the spread reference is relevant, although I don't associate semantic re-use with inheritance (probably because I use inheritance very sparingly). Structural re-use is "I have 3 schemas that already exist, and I need to use them all in this larger schema so I'll allOf them in. But the fact that they are 3 schemas isn't relevant to this larger schema, which would have the same meaning if I copy-pasted all the elements." Semantic re-use might be inheritance, but it also might be delegation. To put it another way, if you're generating anything (code, docs, UI), semantic re-use ought to show up in what you generate. Structural re-use is best hidden as an implementation detail.
• Generics are currently supported via $dynamicRef and $dynamicAnchor. This is part of why runtime references are important, the other being extension/inheritance/whatever with recursive fields (tree structures, for example).
• I'm not sure how a "direct component reference" is an alternative to "inlined in responses", can you elaborate on that a bit?

[handrews]

This topic was discussed extensively in the 20 July 2023 OpenAPI call (see OAI/OpenAPI-Specification#3326 for the agenda and pre-meeting discussion).

My recollection (almost two weeks later) is that:

• We agreed to separate import of documents/resources (using URIs/IRIs) from semantically type-safe referencing
• I argued for using IRIs for better UX for those using non-ASCII alphabets, which I think got general agreement
• I argued for documents to include their own identifiers (like JSON Schema's "$id", but only in one place) to separate identity and location; I don't think there was agreement but it was not rejected out of hand (see below)
• There needs to be some sort of namespacing of imported identifiers, including how to reference identifiers in the current document; I don't remember all of the details of this, and it was clear that more needed to be worked out

Regarding location vs identity: (note: I'll use URI and URL because "IRL" is not a thing) The analogy here is importing modules in a programming language. You import by the module name (its identity - URI), which in some languages has a hierarchical structure and in others does not (or only has a limited one, e.g. NPM "@"-namespaces). The compiler or runtime environment then consults its configuration to find the actual file to load (the location - URL).

We can further extend this analogy by considering package managers, which most major languages have these days. In Python, if I want to use the popular requests module, I first need to install it, which involves downloading it from a server of some sort. So here's what we have:

Identifier (URI-analog): requests

• used to retrieve and install the package: pip install requests
• used to import the package for use in a script or module: import requests

Locations (all of which can be expressed as URLs):

• on first install (from pip logs): https://files.pythonhosted.org:443/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl
• on subsequent installs: (finds it in a cache, although I cannot for the life of me figure out what file that maps to, as it does not appear in pip cache list and the verbose install logs just say it was found)
• when examining the module specification after import requests in interactive Python prompt: file:/Users/handrews/.pyenv/versions/3.8.13/lib/python3.8/site-packages/requests/__init__.py

why should you care?

Back in the old days... (ahem)... installing and using a library was a lot more of a pain. You had to go get the package yourself, and know where to put it. You had to set environment variables or pass command-line flags to the compiler or interpreter.

This is where we are right now with "$ref", except that most people don't understand this analogy and want it to behave like a URL. But you really don't want that.

Look above where the local file is /.../requests/__init__.py. As Python programmers know, that could have just as easily been /.../requests.py if the package was organized differently. You don't have to care, and you shouldn't, you just import requests. Similarly, you don't want to care whether an OAS file you're importing is .json or .yaml or whatever else.

Also, consider if you had to download the package every time you ran a program - not only would performance be horrible, a restrictive firewall (or an air-gapped network) would make it impossible. I've worked at companies that configured internal package manager servers for these reasons.

Plus, sometimes you want to install a development version, which is why pip (and the package managers for many other languages) allow installing from a remote git or other source control repo, or from a local directory.

all of these scenarios apply just as much to working with OAS as working with code

As far as my assertion that OAS documents should declare their own URI/IRI, note that requests declares its own package name for the same reasons. I'm just not showing that because the way it does it is very convoluted, and has to do with the equally convoluted history of Python's packaging systems. Let's not mimic that.

But let's do realize that this is analogous to programming languages, and the software fields has decades of experience learning what is needed for proper tooling with good UX. oascomply will show, in detail, how to accomplish all of that with the current 3.0 approach. But we can make it much nicer in Moonwalk as long as we don't throw away too much out of frustration with URIs vs URLs.