Requests with application/json Content-Type do not correctly assume UTF-8 charset

[eduramiba]

• Framework version: 1.1.1 - 1.1.3
• Implementations: All

Scenario

It seems that the changes for #150 broke the assumption that all JSON is UTF-8, by definition.

Expected behavior

Parsing a JSON request body to a Java type should assume UTF-8 charset.

Actual behavior

It does not correctly handle characters out of ascii set.

Steps to reproduce

Send request with appplication/json in the Content-Type header and strange characters such as » in the JSON body. It will not be correctly parsed and throw a bad request.

If you set the Content-Type header to application/json; charset=UTF-8 then it works fine, but should not be necessary and most REST clients won't do it.

Full log output

Invalid UTF-8 middle byte 0x65\\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 13]\\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream);

Thanks for your support!

[sapessi]

Hey @eduramiba, the reason for the change is that HTTP 1.1 says that the default charset is ISO-8859-1. We could automatically detect UTF-8 and adjust the expected charset if one is not specified in the Content-Type header. However, this would break away from the specs and I'm concerned it may affect existing applications that depend on this behavior and are being ported to the framework. I'm hope to suggestions.

[eduramiba]

I see, but the thing is that in version 1.1 it works just specifying application/json but no longer does after 1.1.1, and it seems almost all REST clients sush as httpie or even Jersey client only send that. Maybe we could add an specific case just for json, when no charset specified, and default to that?

See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean

http://www.iana.org/assignments/media-types/application/json

Thanks for the quick response :)

[sapessi]

Perhaps I could add a field for this in the ContainerConfig object, default it to ISO-8859-1, but give you the ability to change this whenever you want:

LambdaContainerHandler.getContainerConfig()
    .setDefaultContentCharset(StandardCharsets.UTF_8.name())

Thoughts?

[eduramiba]

Yeah I guess that could be a good solution.

[eduramiba]

Will this be included in next release?

Thanks

[sapessi]

That's the plan.

[sapessi]

This is now ready in the core branch for release with the next iteration. You can start using it with the 1.2-SNAPSHOT from the branch @eduramiba.

[eduramiba]

Thanks! @sapessi