Optimizing record-level operations in the SDK

[aaronsteers]

Opening this discussion to explore options for record-level processing optimizations.

Known options

1. Use a faster json processing library.
2. Skip json validation, or provide user options to disable validation for performance benefit.
3. Consider options to perform record-level functions in a lower-level language, either with a library that already uses C/Rust libraries, or by rewriting some of the SDK functions in a lower level language ourselves.

Other throttling factors

As a general rule, only for pipelines that already have an extremely high throughput will benefit at all from record-level python optimizations.

Many taps and targets have performance throttled by factors unrelated to python record parsing.

• Taps often have API rate limiting that throttles how quickly records can be received.
• Taps and targets both have to deal with latency and server-side processing times for the remote systems.
• Targets often have a high cost per batch or a high cost per roundtrip when writing. If the target is not able to keep up with the tap (which is very often the case), then the STDOUT/STDIN buffers will become full and the tap will eventually be paused by the OS in order to let the target catch up. In these cases, performance optimizations on the tap will have basically zero effect on the overall runtime.

In many cases, the above factors actually dwarf the python record processing times. Before assuming that any specific pipelines perf optimizations in Python will have noticeable benefit, the above should be closely evaluated.

Challenges with record-level optimizations

If the tap developer needs to perform custom post_process() operations, these would necessarily be running in Python.

To overcome this challenge, mappings and transformations may need to be defined declaratively, for instance with something like MyStream.ignored_properties = ['ignored_prop1', 'complex.ignored_subprop2'] to remove nodes and MyStream.property_remappings = {'new.node.loc': 'old.node.loc'} to declaratively remap properties. These transformations could then be performed at high speed using a low-level language or highly-optimized library.

Of course, while many taps could benefit from this abstraction, some advanced postprocessing may may be difficult or impossible to abstract in this way.

Alternative approaches

The best alternative to speeding up record-level operations is for the taps and targets to not have to process individual records at all. If the source system can generate a batch files and the target system can read them, then each system is performing in its ideal performance situation, which is not subject to per-record performance considerations on the Python side at all.

See:

• Spec discussion: BATCH message type, first iteration #859