run-time serialization and deserialization of custom formats

[duncanbeevers]

🗣️ Description

I want automatic deserialization + serialization of custom formats; particularly, Date values.

Currently, I get nice type signatures for requests, and for response bodies; however those types are limited to values which can be serialized to JSON. This precludes sending Dates directly to the wire, and reading Dates from responses.

Instead, in my client-side code I often have to create bespoke deserializers which must be used after fetching.

// so many of these…
function deserializeAvailability(
  availability: Availability
): AvailabilityWithDates {
  return {
    ...availability,
    endDate: parseISO(availability.endDate),
    startDate: parseISO(availability.startDate),
  };
}

🎨 Ideal Solution

• openapi-express-validator has a serDes option with a great API for transforming custom values.
• openapi-typescript (using transform) generates serDes-compatible types (eg; createdAt: Date instead of string)
• openapi-fetch has a bodySerializer option, but this operates far too late to make these transformations.

👑 Proposal

• Add a serDes option to openapi-fetch#createClient
• serialize and deserialize custom values in requests and responses

I recognize this is something of a departure from the core philosophy of this package; it's nice to have such a thin run-time, and not have to traverse+transform any data structures.

However, I appreciate the strictness and opinions of openapi-typescript, and want to leverage more of that interpretation throughout the stack.

Perhaps this could be implemented as a pluggable visitor, rather than integrated into the core openapi-fetch library.

I've been mulling this for a little while, and figured I'd at least solicit some feedback about the idea.

Checklist

• I’m willing to open a PR for this (see CONTRIBUTING.md)

[drwpow]

I recognize this is something of a departure from the core philosophy of this package; it's nice to have such a thin run-time, and not have to traverse+transform any data structures.

Not necessarily; I think middleware support (#1122) is still on the table. I just didn’t want to jump into adding that in a way that breaks type safety (it’s hard, for the same reasons your proposal is hard, which I’ll come back to). I’m not opposed to “opt-in slowness” where it’s fast by default, but you can slow it down and opt into runtime transformation if you want to.

• openapi-express-validator has a serDes option with a great API for transforming custom values.
• openapi-fetch has a bodySerializer option, but this operates far too late to make these transformations.

I might be wrong, but it looks like openapi-express-validator actually keeps the OpenAPI schema in memory to do those serializations/deserializations? I think that’d be a non-starter for this library; keeping the entire OpenAPI schema in client memory wouldn’t be possible in many setups (I know this library can be used in Node, but with the browser as the limiting factor we’d need to solve for both, always).

One possible way to solve this would be codegen, maybe via a Vite plugin or something. I’m thinking about how SvelteKit and some GraphQL tools compile types/minimum code automatically in the background as you work. For this, there could be some component of scanning the fetch calls you’re writing (which are statically-analyzable, otherwise the TypeScript inference wouldn’t be working), comparing it against the original schema, and then using that to provide the minimum data that the serializers/deserializers need to do the transforms. Ideally, this would also plug into openapi-typescript so that the correct types are also generated from that, too, without config.

Any codegen utils would probably be a separate package from openapi-fetch, and it’d definitely be experimental for a while as the first attempt would probably be flat-out wrong, and it’d take some iteration to get right.

I know that’s sort of a wild/complex idea, and I glossed over a lot of detail. But all that said, it seems possible? I can’t think of a pure runtime way to accomplish this off the top of my head but will give it some thought.

[duncanbeevers]

might be wrong, but it looks like openapi-express-validator actually keeps the OpenAPI schema in memory to do those serializations/deserializations?

Yes, I believe it does. It doesn't generate type-safe responses, so I use the two tools to complement one-another; one for types, and one for behavior.

I know that’s sort of a wild/complex idea, and I glossed over a lot of detail. But all that said, it seems possible? I can’t think of a pure runtime way to accomplish this off the top of my head but will give it some thought.

I agree an ahead-of-time compilation process makes a lot of sense. The deserializeAvailability I posited above could definitely be generated ahead-of-time and wired-into the fetcher without user intervention.

[duncanbeevers]

I currently use openapi-typescript-fetch as a client library, and pre-generate a file with all the possible operations exported according to their operationId.

import { Fetcher } from 'openapi-typescript-fetch';
export const fetcher = Fetcher.for<paths>();
import { paths } from 'src/generated/admin';

export const availabilityGet = fetcher.path("/availability/{availabilityId}").method("get").create();
export const availabilitiesPost = fetcher.path("/availabilities").method("post").create();
…

When I build the client, it imports from there, and any endpoints not used by the client get tree-shaken away. 🌳 🪓
Although this approach does some unnecessary work, it has a couple of nice properties:

• Signatures for not-yet-used operations are immediately available while a developer is first starting to use them
• A dedicated tool (tree-shaker) is used to trim away unnecessary code; I certainly don't want to write that logic. 😅

[snarky-puppy]

@duncanbeevers that sounds great, how are you generating the file? Would you mind sharing? :-D

[duncanbeevers]

@snarky-puppy It's just a simple traversal+transformation of the schema json.

Start with Object.entries(schema.paths), and everything else falls out from there.

[drwpow]

Going to start exploring this this week. Ideally it involves a third-party library that’s better at this than handrolling something (and is swappable if people don’t like it).

I have a feeling like this will slot into middleware (#1122), but will still need either an example, or maybe even official middleware to guarantee it works well.

[StepanMynarik]

This would be an amazing feature!

We could use this for our date fields for example.

These are defined as string type with date format in API spec.
In our client app we have 3rd party date type called Temporal.PlainDate for extra date calculations etc. Imagine being able to use it directly, without converting it manually everywhere all the time after receiving the raw data from API.

Best thing would be the ability to define custom type transform (string to custom Date type in this case) in combination with openapi-fetch custom functions for both serialization and deserialization of that type. All at the same time/place for maximum type safety.

[moritzruth]

The API of my project makes heavy use of types from the Temporal API. Automatic deserialization would make this (already great) library even more useful.

[github-actions]

This issue is stale because it has been open for 90 days with no activity. If there is no activity in the next 7 days, the issue will be closed.

[StepanMynarik]

This is AFAIK still an issue.

[darkbasic]

You've talked about using middlewares to serialize/deserialize dates, but how do I know which fields to serialize/deserialize without access to either the openapi schema or the typescript ast?

[duncanbeevers]

but how do I know which fields to serialize/deserialize without access to either the openapi schema or the typescript ast?

You are correct that some form of structure must be shipped to the client, but this can be significantly less than the entire schema; only the "special" parts of the response that don't parse natively out of JSON.parse need to considered.

I'll try and write this up in a more-shareable way, but here's what I've done to generate minimal deserializers.

To begin, consider the problem; we have a parsed JSON object, and we would like to selectively rewrite parts of it, taking the primitive value (typically a string) and replacing it with something else.
There are only three types of places where such replacements can happen:

• at the root (eg, replacing the entire response)
• at a named property (eg, replacing user.createdAt = new Date(user.createdAt))
• at an array index (eg, replacing occurrences[0] = new Date(occurrences[0]))

In order to identify the minimal deserializer's shape we can build it up from the schema. My algorithm uses recursive descent, but there are probably smarter ways to implement it.

• iterate over each operation, if it has no JSON response ignore it
• for each operation with a JSON response;
• during the drill-down phase, traverse the response schema, tracking how each sub-schema was reached
  • root response schema
  • property access schema (named properties of object schemas)
  • indexed access schema (items of array schemas)
  • composition (allOf, anyOf, oneOf)
• during the drill-down phase, note whenever we reach a deserializable schema
• during the bubble-up phase, if no deserializable schema was encountered ignore this schema
• during the bubble-up phase, generate the deserializer for this dependency
  • no parent dependency; generate code that replaces the entire (root) response
  • property access dependency of deserializable field; generate code that replaces the named property
  • property access dependency of ancestor field; generate code that traverses the named property
  • indexed access dependency of deserializable field; generate code that replaces all members of the array
  • indexed access dependency of ancestor field; generate code that traverses all members of the array

In my implementation I build the dependency graph first and then run some deduplication passes on the resulting structure, rather than generating code during the traversal, but that's the general idea.