Optional and multi-segment path parameters

[darrelmiller]

This is a meta issue representing a number of requests over the years including #1459 #1840 #892

Currently the OpenAPI specification does not allow optional path parameters nor path parameter values that allow characters such as the /. This means that if you had the following API description:

  /myfiles/{mypath}/{filename}:  
    parameters:
      - name: mypath
        type: string
      - name: filename
        type: string

then you would not be allowed to have myPath = "a/path/to/a/file" unless you escaped the forward slash character.

Also, the RFC6570 URI Template /myreports/{reportName}{/nonDefaultFormat} where the last path segment is optional is not supported by OAS.

The primary reason for not supporting these types of APIs is that it creates the potential of an ambiguous match between a URL and the corresponding path item. Some tooling depends on being able to identify the API description for a specific URL. If multiple pathItems match, then some kind of alternate selection algorithm must be defined.

There have been a number of suggestions on how that selection algorithm might work, with varying levels of complexity. The open question is if there is enough community demand to justify the work necessary to find an acceptable solution.

If you have real-world scenarios where adding support for either multi-segment path parameters or optional path segments would make your life easier, please share them in this issue.

[xuorig]

We have a few use cases when it comes to GitHub API v3. The current workaround is to annotate multi segment parameters with an extension, but this is far from ideal as far as tooling support and specification adherence goes.

• File system / git repository paths. For example creating a file in a repository has a path parameter that takes an arbitrary path: https://docs.github.com/en/rest/reference/repos#create-or-update-file-contents
• Git refs: The ref param in github API can often be a SHA, a branch name, or a tag name. Branch names can easily contain slashes. Example: https://docs.github.com/en/rest/reference/checks#list-check-runs-for-a-git-reference

[jdesrosiers]

The URI Template + operator is another alternative although I think it has the same potential for ambiguity as the / operator. The + operator allows reserved characters, so the example /myfiles/{mypath}/{filename} could be /myfiles/{+mypath}/filename which could match /myfiles/a/path/to/file/myfile to { "mypath": "a/path/to/file", "filename": "myfile"}.

I know @awwright has written a URI matcher for URI Template (https://github.com/awwright/uri-template-router). He might have some insight on how to deal with the ambiguities.

[darrelmiller]

@jdesrosiers The allowReserved property was added to the parameter object in 3.0 to provide the same capability as the + operator, but we limited it to only be usable in query parameters. https://spec.openapis.org/oas/v3.1.0#fixed-fields-9

Yeah, I've travelled that same path writing a URI matcher for URL Templates https://github.com/tavis-software/Tavis.UriTemplates/blob/master/src/UriTemplates/UriTemplate.cs#L355 It's not fun.

[awwright]

uri-template-router tries to invent a general algorithm for reversing a URI Template and it is, indeed, somewhat complicated, in large part because URI Template isn't designed for matching.

The first rule I try to follow is a simple, self-evident principle: If X is a strict subset of Y, then match X before Y.

This means that http://example.com/a/b/ will match templates in the following order:

1. http://example.com/a/{+foo}/ and http://example.com/{+foo}/b/
2. http://example.com/{a}/{b}/ (requires exactly three slash characters in the path)
3. http://example.com/{+a}/{+b}/ (requires at least three slash characters in the path)
4. http://example.com/{+foo}/ (requires at least two slash characters in the path)
5. http://example.com/{+foo} (the biggest set)

Then for picking between disjoint URI templates, prefer templates that are strict subsets when only looking at the first characters. Therefore, http://example.com/a/{+foo}/ would be matched first because if you compare only the first 20 characters of matching URIs, http://example.com/a is a strict subset of http://example.com/{+foo} (where {+foo} matches zero-or-one characters).

Finally, I match empty strings by default. Because URI Template isn't a matching syntax (like regexp), it doesn't have a way for requiring a minimum number of matches. Developers requiring a match are expected to validate the values for the matches, and call Result#next if one of the variables is invalid (e.g. is empty, when a minimum of 1 character is expected).

I recall there being one or two small bugs with the algorithm that I have, I was in the process of mathematically trying to prove that an algorithm implementing all these requirements is computationally possible, and then got sidetracked.

[awwright]

I dove into a "parsing strings" rabbit hole and I've come up with some interesting findings:

A URI Template is a subclass of a regular expression, which in turn can be expressed as a
Non-deterministic Finite Automaton (NDFA), which in turn is expressible as a
Deterministic Finite Automaton (DFA).

There is an optimal, normalized form for DFA. Once you build a DFA from a URI template, you can cache it/save it.

You can reverse the DFA back into a regular expression or URI Template.

Any URI that you can build with a URI Template can be reversed into one or more values for that URI template that will build that URI, and this can be done deterministically (so that there is a "canon" value).

It is also mathematically possible to detect these ambiguous cases, where a single URI is potentially parsed into multiple values.

The algorithms for building the DFA are complicated (at least 1k LoC), but the machine is computationally straightforward: Once you've built the state machine, it can be parsed in O(n) time, n = length of input string, no backtracking.

You can even add complicated processing on top of the URI Template. For example, you could register two URI templates:

1. http://example.com/blog/{+foo} where foo ~= /^\d+$/
2. http://example.com/blog/{bar} where bar ~= /[0-9a-fA-F]+/

... and the DFA compiler could detect that (1) is a subset of (2), because of the regex constraints, even though the URI Templates by themselves would suggest the opposite.

---

In a world where Godel's incompleteness theorem usually rules, and O(n!) algorithms are often our best case, this is pretty exciting. The question is, what can OpenAPI take from this? My opinion is that frameworks exist to remove implementation burden from developers. But writing all of this into OpenAPI (e.g. the regex constraints on URI Template variables) could potentially be prohibitive to complete implementations being written at all.

I'm going to continue to work on this; but what sort of work would you find useful?

[yinzara]

All of APIs that Google provides via protobuf/grpc they also specify a google.api.http option that provides a HTTP 1.0 endpoint. They often use the syntax: /some/prefix/{name=resourceType/*} for path parameters for specific resources. Unfortunately because of the lack of RFC 6570 support in OpenAPI, all of the Google APIs cannot be represented in OpenAPI.

For development teams that choose to support both gRPC and RESTful endpoints, this means they can't follow the guidelines Google provides around API design which saves small teams a huge amount of documentation and maintenance plus gives their APIs a consistent usage.

https://cloud.google.com/apis/design

[ryber]

Hello I would like to offer a real world example of the need for this.

I have a legacy API that uses optional path params for paging information. So URLs like:
https://somewhere/books;count=100?someparam=foo

I'm trying to put a new API gateway in front of this API and the gateway strictly requires all APIs to be done in OpenAPISpec, because I cannot model the optional path param (that I can see), the Gateway is returning 404 for any requests containing it. Unfortunately this is not something I can change, the API has thousands of consumers and serves 40 billion requests a year. Switching to all query params or headers or something else would be a major breaking change. my only other option would be to write some proxy in front of the Gateway to rewrite the URLs but that seems silly. So it would be great if the Spec could support this.

[eugene-bright]

I propose to add support of {+path} expression from RFC6570.
Please vote on this if you agree!

[rafalkrupinski]

wouldn't http://example.com/{+a}/{+b}/ be ambiguous?
Probably only one {+PathElement} per path should be allowed

[awwright]

@rafalkrupinski

wouldn't http://example.com/{+a}/{+b}/ be ambiguous?

No, not really. It's ambiguous in the sense that there's multiple values for a and b that can produce the same URI. However, because these forms all map to the same URI, they must identify equivalent things, so there's no ambiguity in practice.

There's only one match that will be made by a finite state machine; the one such that b will never contain any "/" characters.

[rafalkrupinski]

@awwright

There's only one match that will be made by a finite state machine; the one such that b will never contain any "/" characters.

Isn't that arbitrary? Might as well be parsed such that a never contains any '/' characters.
Besides, that would mean the pattern degenerates to http://example.com/{+a}/{b}/ (or http://example.com/{a}/{+b}/)

[ryber]

In our case we have multiple params per path but they always come at the end:
http://example.com/foo;count=5;start=10