JSON serialization/deserialization performance

[olivere]

JSON serialization/deserialization seems to be an issue with many users, especially running Bulk / Search operations at scale.

Things we could do:

1. Use ffjson for the hot paths like Search/Bulk.
2. Re-using buffers via sync.Pool to reduce GC pressure. Something like this.
3. Keep an eye on #317 and/or use go-drainclose before this gets fixed in Go. Of course, we need to check if this has an impact for Elastic as well.

Related issues #253, #208.

[dimfeld]

In addition to ffjson, I came across https://github.com/mailru/easyjson recently which is a similar concept, using code generation, but purports to be faster. Haven't checked it out in detail though.

[olivere]

Just tested draining the body in PerformRequest (3.) but that doesn't make a difference. If it's a regression in stdlib, it'll be fixed. If it's not, it's safe to keep it as is. Either way, it won't make sense to fix it in Elastic.

[olivere]

This is just an experiment. I don't have an opinion yet if this is a good or bad idea, really. Feedback welcome...

I just tested creating the JSON manually, hard-coded via a bytes.Buffer, in the json-serialization branch. Compare before and after.

This currently only covers the action_and_metadata line (see bulk docs), where JSON serialization can be hard-coded because we know the fields in advance (in contrast to the document data).

For reference, here are the benchmark results (run via go test -run=Bulk -bench=Bulk -benchmem, then compare before/after via benchcmp):

benchmark                                              old ns/op     new ns/op     delta
BenchmarkBulkDeleteRequestSerialization-4              3324          3112          -6.38%
BenchmarkBulkIndexRequestSerialization-4               5754          5710          -0.76%
BenchmarkBulkEstimatedSizeInBytesWith1Request-4        13423         12547         -6.53%
BenchmarkBulkEstimatedSizeInBytesWith100Requests-4     1328849       1213027       -8.72%
BenchmarkBulkUpdateRequestSerialization-4              5009          4641          -7.35%

benchmark                                              old allocs     new allocs     delta
BenchmarkBulkDeleteRequestSerialization-4              29             34             +17.24%
BenchmarkBulkIndexRequestSerialization-4               37             43             +16.22%
BenchmarkBulkEstimatedSizeInBytesWith1Request-4        109            125            +14.68%
BenchmarkBulkEstimatedSizeInBytesWith100Requests-4     10612          12212          +15.08%
BenchmarkBulkUpdateRequestSerialization-4              42             47             +11.90%

benchmark                                              old bytes     new bytes     delta
BenchmarkBulkDeleteRequestSerialization-4              1512          640           -57.67%
BenchmarkBulkIndexRequestSerialization-4               2384          1528          -35.91%
BenchmarkBulkEstimatedSizeInBytesWith1Request-4        6259          3660          -41.52%
BenchmarkBulkEstimatedSizeInBytesWith100Requests-4     631082        371193        -41.18%
BenchmarkBulkUpdateRequestSerialization-4              2440          1520          -37.70%

Applications could further speed up the process and remove JSON serialization altogether if they supply a string or json.RawMessage as Elastic only runs JSON serializer if the data structure cannot be transformed into a string directly (see e.g. bulk_index_request.go).

[r--w]

Hi,
I just tried creating JSON documents manually and passing it as a string. This reduced memory significantly. One thing that is worth mentioning is that you need to escape values as in https://github.com/golang/go/blob/5fea2ccc77eb50a9704fa04b7c61755fe34e1d95/src/encoding/json/encode.go#L788 and use sync.Pool for bytes.Buffer in order to avoid allocations.

[olivere]

@r--w Yes, escaping is required and it is not bulletproof for sure. It's just an experiment to find out if it's worth the effort. I will never implement this for all code paths: Probably only Bulk/Search is critical enough.

[r--w]

In my opinion the option to create JSON manually (during indexing) should be sufficient for all "hardcore" users. Just didn't know that it's avaialble (I should have read the source code before ;)

[olivere]

No. Documentation should be better ;-)

[sidazhang]

@olivere Is this something that you are actively pursuing?

[olivere]

Not at the moment.

There is some performance testing ahead in my day-to-day job, so we'll see how that goes and if I can get the results back into Elastic.

[rami-dabain]

I have used https://github.com/buger/jsonparser, it's about 10X faster than stock json package

[olivere]

@rami-dabain Do you have any stats on how much time is actually spend in JSON encoding/decoding for your use case? Can you post your code somewhere?

[olivere]

A pattern I recently use is to split work into several goroutines via a pipeline. The excellent golang.org/x/sync/errgroup makes that dead simple. Here's an example of that pattern.

JSON performance might not be perfect in stdlib. However, if you use all cores, you might not need
to squeeze out the last nanoseconds. With the example above, I was able to saturate our Elasticsearch cluster, and suddenly JSON decoding performance seems like a minor issue. ;-)

I'm very interested in getting feedback. Does that work for you as well?

[rami-dabain]

I havent used jsonparser directly with ES. However I used it in an HTTP receiver service, CPU was maxed out at around 12k requests/second (as the receiver was decoding the json-request), that went up to ~45k requests/second when I switched parsing the json using the mentioned library, most gains were on fetching values for a known key-path. I suggest giving it a try as it doesn't require any "compilation"/code-generation

[olivere]

I'm not going to use jsonparser in Elastic by default. But what could possible work is to return structures (for performance critical services) in a way that users are able to use alternative JSON parsers like jsonparser.