Codable support for APIGateway V2 request and response payloads

[eneko]

Hi there!

I'm trying to update APIGateway.V2 request/response so their body can be any payload conforming to Codable.

Since API requests can be many formats other than JSON, I've added APIGateway.V2.JSONRequest and APIGateway.V2.JSONResponse as separate, generic, types that can be used on Lambda function handlers as In/Out. These are basically a copy of APIGateway.V2.Request and APIGateway.V2.Response, but made generic with a Codable body property.

For example, given the following Request and Response payloads:

struct Request: Codable {
    let name: String
}

struct Response: Codable {
    let message: String
}

We could use them with AWS API Gateway V2 as follows:

// MARK: - Run Lambda

Lambda.run(APIGatewayProxyLambda())

// MARK: - Handler, Request and Response

struct APIGatewayProxyLambda: EventLoopLambdaHandler {
    typealias In = APIGateway.V2.JSONRequest<Request>
    typealias Out = APIGateway.V2.JSONResponse<Response>

    func handle(context: Lambda.Context, event: In, callback: @escaping (Result<Out, Error>) -> Void) {
        let message: String
        if let name = event.body?.name {
            message = "Hello, \(name)!"
        } else {
            message = "Hello, World!"
        }
        let payload = Response(message: message)
        callback(.success(APIGateway.V2.Response(statusCode: .ok, body: payload)))
    }
}

This code, while it compiles fine, is giving me an error when invoking the function. Here is what I see in CloudWatch:

2020-06-14T15:17:36+0000 warning: lifecycleIteration=0 lambda handler returned an error: requestDecoding(Swift.DecodingError.typeMismatch(Swift.Dictionary<Swift.String, Any>, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "body", intValue: nil)], debugDescription: "Expected to decode Dictionary<String, Any> but found a string/data instead.", underlyingError: nil)))

Thoughts? I think having Codable support for API Gateway V2 payloads is a big plus.

Hopefully we can get it to work, any help would be appreciated.

[swift-server-bot]

Can one of the admins verify this patch?

[swift-server-bot]

Can one of the admins verify this patch?

[swift-server-bot]

Can one of the admins verify this patch?

[swift-server-bot]

Can one of the admins verify this patch?

[fabianfett]

Hi @eneko, thanks for bringing this up and providing a PR.

This code, while it compiles fine, is giving me an error when invoking the function.

The problem you're running into is the following. The body is by default encoded as un utf8 string, even if it contains json. That mean's you can not decode the whole structure in one go anyway.

The json you receive looks like this:

{
  "otherProperty": "foo",
  "body": "{\"some\":\"json\"}",
  "otherProperty": "bar"
}

As you can see the body is a String in which another json struct is encoded. You need to read the String first, create a new json data blob Data(body.utf8) and only then you can use a new or the existing JSONDecoder to decode your Payload.

While you raise a valid point, I'm unsure if copy and pasting the existing APIGateway.V2.Request and APIGateway.V2.Response is the best approach here.

Personally I'd prefer to see extensions on the original event types that provide such functionality. If the original event types need to be open up for those extensions I'm all for it.

[eneko]

Oh, that makes total sense, thank you @fabianfett!

I agree 100%. Knowing how the data is structure, providing a Codable getter for APIGateway.V2.Request, and a Codable initializer for APIGateway.V2.Response as extensions seem a much cleaner approach.

I've updated this PR with these changes, let me know what you think.

[eneko]

I've added some unit tests and inline documentation. If this looks like a good approach, I'll move the PR to "Ready for Review"

[GalCohen]

how about:
public func decodedBody<Payload: Codable>(decoder:JSONDecoder = JSONDecoder()) throws -> Payload? {

This way the user has can customize the decoder and provide their own. For example, to change the dateDecodingStrategy.

[GalCohen]

(Same suggestion for the JSONEncoder usage below)

[ktoso]

That's a good point but also yet another instance where "TopLevelDecoder" types would be the right thing to go with... Since we'd like to allow using various coder implementations, not marry the API to Foundation's impl only.

We could do them ad-hoc here in the lambda lib again as Combine does, but that's growing the problem a bit; Or we ignore and go along with this for now, but eventually try to resolve it once we get to it.

// fyi @tomerd

[eneko]

Thanks for the feedback. I've updated the PR to inject both JSONEncoder and JSONDecoder.

I agree a "TopLevelDecoder" type would be ideal, let me know if I should do any further changes.

[ktoso]

Another idea would be to see if it's possible to push the functions out into an "AWSLambdaFoundationCompat" module or something similar?

[eneko]

@ktoso I was looking at Lambda+Codable.swift from AWSLambdaRuntime and it seems like encoding/decoding work could be moved to a separate module, like you suggest. I haven't tried it, though.

Would you be open for that work being done on a separate pull request? While I think it is valuable, it feels out of the initial scope of this work.

[GalCohen]

Sorry, I don't know the internals here, based on a github search I don't see too many instances of "import Foundation". Would that mean that the lambda runtime wouldn't depend on Foundation? That might be nice for performance, right?

[eneko]

@calcohen, at the moment, yes, AWSLambdaRuntime depends on Foundation for Codable support:

swift-aws-lambda-runtime/Sources/AWSLambdaRuntime/Lambda+Codable.swift

Line 17 in 246088e

import class Foundation.JSONEncoder

It might be possible to remove this dependency, like @ktoso suggested, but to me that feels like work for a different pull request.

[tomerd]

typo

[tomerd]

rename Payload -> Body

[tomerd]

I wonder if instead of decoder: JSONDecoder = JSONDecoder() which should do something like decoder: (Body) -> String and have a default impl that uses JSONDecoder

[tomerd]

following on that line of thinking, @fabianfett maybe knows if this should be (Body) ->[UInt8], or String is good enough

[eneko]

The decoder should be either (String) throws -> Body or (String.UTF8View) throws -> Body, since we are decoding the content of the body property. Will update.

[eneko]

I've given this a shot, but kind of defeats the purpose of the pull request.

Here is what the extension would look like:

public func decodedBody<Body: Codable>(decoder: (String.UTF8View) throws -> Body) throws -> Body? {
    guard let utf8 = body?.utf8 else {
        return nil
    }
    return try decoder(utf8)
}

And what the callee would look like:

let decoder: (String.UTF8View) throws -> Body = { utf8 in try JSONDecoder().decode(Body.self, from: Data(utf8)) }
let body: Body? = try request.decodedBody(decoder: decoder)

At that point, using this extension would yield close to no benefit.

[tomerd]

@eneko I meant the library would offer a default impl as well, for example (non optimized):

public func decodedBody<Body: Decodable>(decoder: @escaping (String) throws -> Body = Self.jsonDecoder) throws -> Body? {
  try self.body.map(decoder)
}
        
public static func jsonDecoder<T: Decodable>(json: String) throws -> T {
  try JSONDecoder().decode(T.self, from: Data(json.utf8))
}

[tomerd]

rename Payload -> Body

[tomerd]

tbd.. same comment like the decoder above

[tomerd]

@swift-server-bot test this please

[fabianfett]

Hi @eneko,

if I understand you correctly, your primary goad is to have a one liner to get the json payload out of the APIGateway.V2.Request. To be honest we also have the same problem with SQS.Message, SNS.Message and APIGateway.Request.

The real issue imho is that people can't just use the JSONDecoder to decode strings right away. One thing that would solve this is to make the request decoding a one liner, could be an approach like this:

// MARK: - Run Lambda

Lambda.run(APIGatewayProxyLambda())

// MARK: - Handler, Request and Response

struct APIGatewayProxyLambda: EventLoopLambdaHandler {
    typealias In = APIGateway.V2.JSONRequest
    typealias Out = APIGateway.V2.JSONResponse

    struct Request: Codable {
        let name: String
    }

    struct Response: Codable {
        let message: String
    }

    func handle(context: Lambda.Context, event: In, callback: @escaping (Result<Out, Error>) -> Void) {
        do {
            let jsonRequest = try self.decoder.decode(Request.self, from: event.body)
            // ^-- magic here
        } catch {
            return .success(.init(status: .badRequest))
        }
        // unimportant right now
    }
}

There are other JSON libraries out there with which you can do this out of the box, since they use Collections of UInt8 as the input.

Please also note that we use the default decoder of the lambda handler to increase performance.

So I propose the following we add an extension to JSONDecoder in the AWSLambdaRuntime target. AWSLambdaRuntime is the place for all Foundation JSON stuff anyway.

extension JSONDecoder {
    func decode<T: Decodable>(_ type: T.Type, from string: String) throws -> T {
        guard let data = Data(from.utf8) else {
            preconditionFailure("must never happen")
        }
        return try self.decode(type, from: data)
    }
}

This way we would add more convenience to JSONDecoder without moving the implementation of AWSLambdaEvents into a more Foundation biased direction.

For the encoder we could go into a similar direction with the encoder supporting encoding into a String.

@eneko would this work for you?

@tomerd @ktoso wdyt?

[eneko]

Hi @fabianfett,

From what I see, that solution could work well. I agree using the default encoder/decoder form AWSLambdaRuntime would be great. Whether it uses Foundation or something else, it shouldn't matter to the end user.

Before getting further down this path, let's take a step back. I think it is important to clarify the goal, so we can find the best solution.

if I understand you correctly, your primary goad is to have a one liner to get the json payload out of the APIGateway.V2.Request

I have a bit higher expectations: the real goal, in my opinion, would be to have a zero-liner solution. Let me explain.

This is the most basic Codable example, from @tomerd blog post:

private struct Request: Codable {
  let name: String
}

private struct Response: Codable {
  let message: String
}

// In this example we are receiving and responding with JSON using Codable
Lambda.run { (context, request: Request, callback) in
  callback(.success(Response(message: "Hello, \(request.name)")))
}

The simplicity of this code is great. Codable lets users focus on their data transfer models, instead of manually having to deal with encoding, decoding.

We should be able to provide the same level of simplicity when working with any type from AWSLambdaEvents, as long as the payload is JSON.

To be honest we also have the same problem with SQS.Message, SNS.Message and APIGateway.Request

I'm not sure about SQS.Message and SNS.Message. However, for APIGateway and APIGateway.V2 we should be able to look at the request headers. Then, if the content-type of the request is application/json, automatically decode the body to a given Body type using generics.

I'm hoping I'm making sense here. The goal, to summarize, would be to automatically decode the body payload, when content type is JSON, and provide this decoded body to the lambda handler method, so users do not have to worry about it.

Rewriting the code above using APIGateway.V2, this is what I would want the API to look like:

private struct Request: Codable {
  let name: String
}

private struct Response: Codable {
  let message: String
}

// In this example we are receiving and responding with JSON using Codable
Lambda.run { (context, event: APIGateway.V2.Request<Request>, callback: @escaping (Result<APIGateway.V2.Response<Response>, Error>) -> Void) in
  let body = Response(message: "Hello, \(event.body.name)")
  callback(.success(APIGateway.V2.Response(statusCode: .ok, body: body)))
}

This code would need some tweaks, probably to handle encoding/decoding errors, but maybe we could abstract that too.

Again, the goal would be to have a fully decoded body prior to invocation of the handler.

This should be viable. It might require exposing AWSLambdaRuntime encoder/decoder, so they can be used by AWSLambdaEvents. Or maybe, as suggested, AWSLambdaEvents could use a generic interface provided by the runtime module, that abstracts this encoding/decoding. In any case, the runtime module would have to expose an API for encoding/decoding JSON (or other types of payload encoders/decoders, like XML).

Let me know if the makes sense. I appreciate each comment on this review, great feedback, I'm learning a lot.

[tomerd]

So I propose the following we add an extension to JSONDecoder in the AWSLambdaRuntime target. AWSLambdaRuntime is the place for all Foundation JSON stuff anyway.

I think this would be a useful and nice addition

Again, the goal would be to have a fully decoded body prior to invocation of the handler.

I tend to agree this would be the optimal user experience

[fabianfett]

@eneko

Again, the goal would be to have a fully decoded body prior to invocation of the handler.

@tomerd

I tend to agree this would be the optimal user experience

I'm sure such an EventType could be build. I'm not sure though if this should be part of the more general focused AWSLambdaEvents in such a case.

In general purpose frameworks this task would require some registering of BodyParsers that then emit create types. This should imho be done as part of a framework that builds on top of AWSLambdaRuntime that should also offer routing.

[tomerd]

@enemo just fyi: not forgotten this PR, hope to share a proposal soon

[tomerd]

Hi @eneko, thank you for you patience. @fabianfett and myself discussed and experimented with various alternatives, and would like to bring up some of the ideas and potential future directions for wider discussion.

The desire to have a strongly typed lambda function clearly resonates. Being able to define a Request<Body> and Response<Body> in API Gateway handlers as an example and have the system take care of the encoding/decoding is clearly a nice user experience. The challenge, of course, is in the implementation details.

As you mention, several AWS event payloads provide a free-from body: e.g. API Gateway, SNS, and SQS.
Those payloads do not say much about the serialization of that body - they provide the content as a String and an isBase64Encoded flag so that the consumer can decode based on the expected format.

The body could be JSON, and its often is, but it could also be any other format e.g. XML, or a binary blob encoded as base64. As such, in order for the lambda runtime library to utility for body encoding/decoding it's not enough to know the expected type, but we also need to know the expected encoder/decoder. Of course, we can default to JSON to make the API easier, but we need to allow extension points to configure alternatives.

Following this line of thinking, the only generic implementation option we could come up with would be to define a new type that represents the body and provide easy-to-use encoding/decoding APIs that default to JSON. For example (illustration purposes only):

public struct AWSBody: RawRepresentable, Equatable {
    public let rawValue: String
    public init?(rawValue: String) {
        self.rawValue = rawValue
    }
}

extension AWSBody {
    public func decode<Body: Decodable>(_ type: Body.Type, decoder: (String) throws -> Body = Self.defaultDecoder) throws -> Body {
        try decoder(self.rawValue)
    }
    public static func defaultDecoder<Body: Decodable>(string: String) throws -> Body {
        return try Lambda.defaultJSONDecoder.decode(Body.self, from: string.data(using: .utf8)!) // safe since String.data should never return nil
    }
}

extension AWSBody {
    public init<Body: Encodable>(_ body: Body, encoder: (Body) throws -> String = Self.defaultEncoder) throws {
        self.init(rawValue: try encoder(body))! // safe since AWSBody::init never returns nil
    }
    public static func defaultEncoder<Body: Encodable>(body: Body) throws -> String {
        return try String(data: Lambda.defaultJSONDecoder.encode(body), encoding: .utf8)! // safe since String(data:) from JSON should never return nil
    }
}

This means the API will not end up being the desired Request<Body> and Response<Body> but rather payload.body.decode(Body.self) which arguably is nicer than JSONDecoder().decode(body, Body.self) but does not reduce the number of lines of code compared to what we have right now. On the other hand, this approach does enable us to add additional utility around base64 handling and users could extend it further to their use case.

Another approach could be to create a dedicated LambdaHandler that takes in the body type an associatedtype and has injection points for the encoder/decoder. For example (illustration purposes only):

protocol APIGatewayLambdaHandler: EventLoopLambdaHandler {
    associatedtype Request: Decodable
    associatedtype Response: Encodable
    func handle(context: Lambda.Context, request: Request?) -> EventLoopFuture<Response>
    func encode(response: Response) throws -> String?
    func decode(request: String) throws -> Request
}

extension APIGatewayLambdaHandler {
    func handle(context: Lambda.Context, event: APIGateway.V2.Request) -> EventLoopFuture<APIGateway.V2.Response> {
        let requestBody: Request?
        do {
            requestBody = try event.body.map(self.decode)
        } catch {
            return context.eventLoop.makeFailedFuture(error)
        }
        return self.handle(context: context, request: requestBody).flatMapThrowing { response in
            let responseBody = try self.encode(response: response)
            return APIGateway.V2.Response(statusCode: .ok, body: responseBody)
        }
    }
}

extension APIGatewayLambdaHandler {
    func encode(response: Response) throws -> String? {
        try String(data: Lambda.defaultJSONEncoder.encode(response), encoding: .utf8)
    }
    func decode(request: String) throws -> Request {
        try Lambda.defaultJSONDecoder.decode(Request.self, from: request.data(using: .utf8)!)
    }
}

This approach is much more heavy weight but has the advantage of setting the foundation for providing additional functionality around certain event types. Its especially compelling for api-gateway integration where its common to need other middleware-like functionality such as routing. The challenge with this approach is how far we want to extend the runtime library, or IOW should we leave this kind of more opinionated functionality to libraries that the community builds on top of the runtime library?

Of course the approaches are not mutually exclusive - the second can be built on top of the first.

WDYT?

[eneko]

This is great, @tomerd, thanks for getting back. Personally, I like the second approach, with associated types. It looks pretty similar to other code I've worked with, were we have Payload, Response and ErrorResponse associated types for network requests.

Probably at this point, the best option is to make this a separate Swift Package. Then, we could merge it into this one if it had enough support and met all needs/requirements. I'd be happy to work on that, but might take some time.

[fabianfett]

Closing in favor: #144