Take exception message from OpenAPI document

[bkoelman]

Today, when a non-successful status code is returned (which is described in the OpenAPI document), an exception is thrown with an unhelpful generic message:

Exception of type 'MyProject.Models.ErrorResponseDocument' was thrown.

In contrast, NSwag takes the message from the OpenAPI document.

For comparison, here are my test assertions from Kiota:

var exception = (await action.Should().ThrowExactlyAsync<ErrorResponseDocument>()).Which;
exception.ResponseStatusCode.Should().Be((int)HttpStatusCode.Conflict);
exception.Message.Should().Be($"Exception of type '{typeof(ErrorResponseDocument).FullName}' was thrown.");
exception.Errors.ShouldHaveCount(1);

Compared with those from NSwag:

var exception = (await action.Should().ThrowExactlyAsync<ApiException<ErrorResponseDocument>>()).Which;
exception.StatusCode.Should().Be((int)HttpStatusCode.Conflict);
exception.Message.Should().Be("HTTP 409: The request body contains conflicting information or another resource with the same ID already exists.");
exception.Result.Errors.ShouldHaveCount(1);

Here's the OpenAPI fragment the code was generated against:

"responses": {
  "400": {
    "description": "The query string is invalid or the request body is missing or malformed.",
    "content": {
      "application/vnd.api+json": {
        "schema": {
          "$ref": "#/components/schemas/errorResponseDocument"
        }
      }
    }
  },
  "409": {
    "description": "The request body contains conflicting information or another resource with the same ID already exists.",
    "content": {
      "application/vnd.api+json": {
        "schema": {
          "$ref": "#/components/schemas/errorResponseDocument"
        }
      }
    }
  },
  "422": {
    "description": "Validation of the request body failed.",
    "content": {
      "application/vnd.api+json": {
        "schema": {
          "$ref": "#/components/schemas/errorResponseDocument"
        }
      }
    }
  }
}

And, for reference, our shared ApiException class we point NSwag to, where we format the exception message by prefixing it with the HTTP status code (note: the constructor signatures are prescribed by NSwag):

// We cannot rely on generating ApiException as soon as we are generating multiple clients, see https://github.com/RicoSuter/NSwag/issues/2839#issuecomment-776647377.
// Instead, we configure NSwag to point to the exception below in the generated code.

namespace JsonApiDotNetCore.OpenApi.Client.NSwag;

public class ApiException(string message, int statusCode, string? response, IReadOnlyDictionary<string, IEnumerable<string>> headers, Exception? innerException)
    : Exception($"HTTP {statusCode}: {message}", innerException)
{
    public int StatusCode { get; } = statusCode;
    public virtual string? Response { get; } = string.IsNullOrEmpty(response) ? null : response;
    public IReadOnlyDictionary<string, IEnumerable<string>> Headers { get; } = headers;
}

public sealed class ApiException<TResult>(
    string message, int statusCode, string? response, IReadOnlyDictionary<string, IEnumerable<string>> headers, TResult result, Exception? innerException)
    : ApiException(message, statusCode, response, headers, innerException)
{
    public TResult Result { get; } = result;
    public override string Response => $"The response body is unavailable. Use the {nameof(Result)} property instead.";
}

So, to summarize, would it be possible for Kiota to use the text from the OpenAPI document as the Exception.Message value of the generated exception class? It should be a parameter (like in NSwag), because the same exception type could be used for multiple status codes (each with a different message).

[baywet]

Thanks for the suggestion @bkoelman!

To add some context to that: kiota was designed to get the exception message from the response payload.
But in some cases the response payload is missing, not documented to have a primary error message, or cannot be deserialized (not schema, content type not supported).
In those cases, it could default the message to what's in the description.
We already have the infrastructure to model that, the description message could be modeled as a code constant.

Is this something you'd be willing to submit a pull request for?

[bkoelman]

Perhaps in the future, but not anytime soon.

It would also be helpful to have an out-of-the-box default implicit handler for HTTP 500, if that's not in the OpenAPI document. It should read the first few kb of the response body and use that for the exception message, if unable to match the component schema or the content type is incompatible.

The reason is that it feels odd to add 500 as an expected outcome for every endpoint. And even if I did, I can't guarantee the response structure. This would mean a world of difference in complex systems that just log exception.ToString() as a fallback. Nswag does that by default.

[baywet]

This second suggestion could have serious performance impacts in some scenarios. I don't think we'll implement that.

As for the default handler, you can use code XXX (as opposed to 503 or 5XX) to do that today already.
We're also thinking about implementing overlays so you don't have to maintain a copy of the description

[bkoelman]

Can you elaborate on the performance impact? I would have thought reading the first few kb avoids that.

[bkoelman]

Well, I tried adding a 5XX response, but that's a complete disaster:

{
  "responses": {
    // ....
    "400": {
      "description": "The query string is invalid.",
      "content": {
        "application/vnd.api+json": {
          "schema": {
            "$ref": "#/components/schemas/errorResponseDocument"
          }
        }
      }
    },
    "5XX": {
      "description": "Unexpected error",
      "content": {
        "text/plain": {}
      }
    }
  }
}

This makes kiota generate string? as the method return type (so for all status codes, including success). Same when I replace 5XX with default.

Then I tried adding the content type and component schema:

{
  "responses": {
    // ....
    "400": {
      "description": "The query string is invalid.",
      "content": {
        "application/vnd.api+json": {
          "schema": {
            "$ref": "#/components/schemas/errorResponseDocument"
          }
        }
      }
    },
    "5XX": {
      "description": "Unexpected error",
      "content": {
        "application/vnd.api+json": {
          "schema": {
            "$ref": "#/components/schemas/errorResponseDocument"
          }
        }
      }
    }
  }
}

Which does work properly. But changing "5XX" to "default" generates error mappings for "4XX" and "5XX", which is wrong. It should catch any other status code, not just errors.

But this is all just for fun. I'm not going to add content-type and schema for "something unexpected happened", because, like I said, I can't guarantee any format or structure of the response body in such cases. And I'm not even going to add "5XX" or "default" at all, because the OpenAPI spec tells me it shouldn't be done:

Note that an API specification does not necessarily need to cover all possible HTTP response codes, since they may not be known in advance. However, it is expected to cover successful responses and any known errors. By “known errors” we mean, for example, a 404 Not Found response for an operation that returns a resource by ID, or a 400 Bad Request response in case of invalid operation parameters.

---

At this point, I'm very disappointed by kiota. Not only have I come to realize it's still very immature, but more importantly, critical blockers are not being addressed. So far, I've only built simplistic demos (basically what a developer advocate does, nowhere near real-world usage) and already found 8 problems, for which I had to implement ugly workarounds. Users are already expressing concerns about kiota support while we haven't even released anything. Four months ago, when I wrote:

So I thought I'd give Kiota a try. Its code looks clean, is documented and the project is active. Using query strings was just the first issue I ran into, and I doubt it will be the only one. I'm curious if we can work together and find ways to address them. If all goes well, I'd be happy to drop support for NSwag in favor of Kiota, but it's too early for such a decision.

Absolutely.

I had something else in mind than you guys asking me to create a PR all the time. No? Then leave it lingering on the backlog forever. One time, I even got an issue assigned, before getting the chance to respond, pretty brutal. I'm not responsible for fixing your bugs and addressing its deficiencies.

When I was still studying, I worked part-time at a supermarket. I remember the full-time employees repeatedly getting frustrated by the impossible floor plans they'd get from the head quarters. Clearly, those "people at the office" had never actually visited a supermarket. I've built webbased enterprise systems for 20 years using .NET, but switched to the tech sector (building libraries like kiota) a few years ago. Likewise, I find it stunning how little "tech-sector" developers know or care about building actual user-facing systems. Seeing that from the inside, I realized there are plenty of examples within Microsoft too. Where teams are building things primarily for themselves, not what customers ask for. This got worse when Microsoft open-sourced projects and gave teams more freedom. Looking at the open/closed kiota issues, there's clear user feedback: core docs/samples are missing, versioning is a mess, lack of csproj integration (inner loop), handling unstructured data, logging, error handling, warnings we can't fix (so we'll just suppress all), extensibility, nullability (by the way, the top-voted issue). The theoretical approach to "just ensure your OpenAPI document is right and everything is perfectly modeled" shows lack of real-world experience. We can't always change it, nor do we want to maintain a "diff layer" that compensates for its imperfections, just to satisfy rigid kiota. What we need is easy ways to break out when the world isn't perfect. If that means creating an opportunity for people to do something wrong sometimes, or take a little performance hit sometimes, that's a sound tradeoff.

[baywet]

Can you elaborate on the performance impact? I would have thought reading the first few kb avoids that.

The first few kbs is a very arbitrary definition which will lead to people wanting the full thing. And if we provide the full thing by default, a large response could lead to memory pressure, network IOs and even a herd stampede in distributed systems when things are starting to fail.

This makes kiota generate string? as the method return type (so for all status codes, including success). Same when I replace 5XX with default.

I'm not sure how you got a string as a return type as a consequence of changing the error type. A schema like this one will be ignore as it's empty though.

Which does work properly. But changing "5XX" to "default" generates error mappings for "4XX" and "5XX", which is wrong. It should catch any other status code, not just errors.

Default was a design mistake on the specification side. Although it can be handy to save time to the description author, it's useless to most other audiences: is this an error, is this successful, what expectations does the service have on the client behaviour, etc... We made the opiniated choice that default means "error" as it's the broader usage we've seen out there, knowing perfectly that it wouldn't suit everybody. Our guidance is: don't use default. But overall yes, document very specific error codes that are known in advance, and if the service has a generic structure for errors, use the code class mapping (4XX, 5XX) as a shorthand.

On the last 3 paragraphs, feedback taken. We received similar feedback recently at an internal event. Looping in @sebastienlevert and @maisarissi our PMs to highlight this as further evidence for prioritization of the work.

[darrelmiller]

@bkoelman I agree that having to add an explicit 5XX for every operation is a pain and shouldn't be necessary. But also, if you don't then you can't provide a custom error message. OpenAPI v3.x doesn't support defining responses at the global scope. We have proposed that for moonwalk though https://github.com/OAI/sig-moonwalk?tab=readme-ov-file#responses. Also, there is a proposal for the Overlays specification https://github.com/OAI/Overlay-Specification/pull/17/files?short_path=2df9558# that makes this easier. We are currently planning to add Overlay support to Kiota.

I'm sorry that you are disappointed with your experience. We appreciate your support to date. Fully supporting JSON:API would be awesome for Kiota, however, because it is a mature and opinionated protocol, I'm not surprised that we are running into scenarios that we don't currently support. We do not see much usage of JSON:API in our regular community because of the overlap with OData.

The effort to make this a fully OSS project does not come from Microsoft the organization, but from a few Microsoft employees that wanted an OpenAPI generator that fully supported the OpenAPI specification across a broad range of languages. We recognize that there is lots more work to do, but we have other responsibilities as well, beyond contributing to this project, and so it may feel like progress is slow sometimes. However, we are working to ensure that this project is a long term effort that Microsoft will continue to fund and so that we can reach the level of quality we all aspire too across a broad range of languages.

[bkoelman]

Can you elaborate on the performance impact? I would have thought reading the first few kb avoids that.

The first few kbs is a very arbitrary definition which will lead to people wanting the full thing. And if we provide the full thing by default, a large response could lead to memory pressure, network IOs and even a herd stampede in distributed systems when things are starting to fail.

You're just making my point. Rejecting to solve a practical limitation, for theoretical reasons, which makes developers hate using kiota. Obviously, the team would never agree to fetch the full response, and for good reasons. But nobody is asking for that. So I looked it up: NSwag takes the first 512 bytes. It was implemented 10 years ago and worked well since. Why ignore proven solutions and reinvent the wheel all the time? I find it deeply concerning that I even need to explain this. So, just start with 512 bytes. In the unlikely case you get massive pushback, consider bumping that to 1 kb.

Here's a practical example: I'm debugging a complex solution with many services and layers, then hit an unhandled exception in the IDE. Right now this gives me useless information:

Exception of type 'MyProject.Models.ErrorResponseDocument' was thrown.

So now I need to guess where to add breakpoints (assuming it's easily reproducible). Or dive into log files, correlate requests, etc. While it should just be as simple as this:

> HTTP 500: Unexpected error. First 512 bytes of response body:
> 
> System.InvalidOperationException: The binary operator GreaterThan is not defined for the types 'System.String' and 'System.String'.
   at System.Linq.Expressions.Expression.GetUserDefinedBinaryOperatorOrThrow(ExpressionType binaryType, String name, Expression left, Expression right, Boolean liftToNull)
   at System.Linq.Expressions.Expression.GreaterThan(Expression left, Expression right, Boolean liftToNull, MethodInfo method)
   at JsonApiDotNetCore.Queries.QueryableBuilding.WhereClauseBuilder.VisitComp

This makes kiota generate string? as the method return type (so for all status codes, including success). Same when I replace 5XX with default.

I'm not sure how you got a string as a return type as a consequence of changing the error type. A schema like this one will be ignore as it's empty though.

Well, just try it. This is what I observed.

Which does work properly. But changing "5XX" to "default" generates error mappings for "4XX" and "5XX", which is wrong. It should catch any other status code, not just errors.

Default was a design mistake on the specification side. Although it can be handy to save time to the description author, it's useless to most other audiences: is this an error, is this successful, what expectations does the service have on the client behaviour, etc... We made the opiniated choice that default means "error" as it's the broader usage we've seen out there, knowing perfectly that it wouldn't suit everybody. Our guidance is: don't use default. But overall yes, document very specific error codes that are known in advance, and if the service has a generic structure for errors, use the code class mapping (4XX, 5XX) as a shorthand.

Then why did you suggest it? I didn't even know it was possible until you told me to:

As for the default handler, you can use code XXX (as opposed to 503 or 5XX) to do that today already.