Performance benchmark

[vaind]

We should have a set of benchmarks to check the performance of the current implementation and also to relate to when changes are made (for example when #68 is implemented). In short, the benchmark should address CRUD performance, bulk operations (putMany()) and queries.

You can take clue from other ObjectBox language bindings:

• objectbox-go
• objectbox-java

Also, have a look at these blog posts:

• https://objectbox.io/how-to-benchmark-database-performance-and-objectbox/
• https://objectbox.io/go-1-0-release-and-performance-benchmarks/

[vaind]

Looks like Dart DevTools is the tool to use. If I'm not missing something, it's a little impractical because you have to start it while the program is running and can only use it while it hasn't stopped. So in order to profile performance, I'm doing the following:

• add a breakpoint before and after an interesting section
• start the program
• start DevTools after the program hits the breakpoint
• hit "Record" button in the "Performance" tab in DevTools
• resume the program, let it continue to the next breakpoint
• hit "Stop" button in DevTools
• evaluate the generated performance chart

Rough CRUD benchmark (10k objects, mass ops, see related PR)

Function	Runs	Average ms
putMany		30	110
readAll		30	49
updateAll	30	112
removeAll	30	7

The numbers are a little underwhelming, compared to ObjectBox performance in other languages so I'm going to look for potential improvements next.

[greenrobot]

add a breakpoint before and after an interesting section

Do I correctly assume you used the profile mode?

• For reference, please post the CRUD results for all 3 modes (debug, profile, release).

As for the first profiling it seems that marshalling/unmarshalling is the major bottleneck. This is expected; especially since we did not address #68 yet (generate code to directly create FB from dart objects and the reverse). Do the profiling results confirm that #68 should have most impact? Either way, I'm lacking some info on the "mapping" from profiling results to optimizations.

[vaind]

Do I correctly assume you used the profile mode?

Nope, that's flutter only AFAICT. Doesn't do anything for dart CLI apps (doesn't even affect the build/run command executed by the IDE). The results above are coming from Dart VM run (i.e. hitting run in an IDE or running dart main.dart). There's also dart2native which creates a standalone program with a little higher performance but you can't profile that.

Function        Runs    Average ms  
putMany          30      90
readAll          30      41
updateAll        30      87
removeAll        30      7

Do the profiling results confirm that #68 should have most impact?

Nope, copying memory from flatbuffers to C does. See how much OBX_bytes_wrapper.managedCopyOf() takes out of the marshal() call or the whole putMany() for that matter. It's already addressed in the PR #164

[greenrobot]

The results above are coming from Dart VM run

And this is... release mode? At least the difference to "native", while significant, is not super bad.

OBX_bytes_wrapper.managedCopyOf() ... It's already addressed in the PR #164

Time to do have another profiling session then? I'm really hoping ffi overhead is less than our own sub-optimal handling (#68).

And what's that _loadStruct bottleneck for reading?

[vaind]

And this is... release mode?

🤷 no such things in dart. Kinda looks like "dart VM" is for dev and debugging, while compiling with dart2native should yield the best performance... but that's all guesswork and not something I wanted to investigate since I can only profile when running in VM anyway. Also Google search didn't turn out anything and dart command doesn't seem to have any related options. We can (later) run the benchmark from a flutter app in modes available there to see how those relate.

Time to do have another profiling session then?

I'll post profiler screenshots for state after each commit in #164

And what's that _loadStruct bottleneck for reading?

That's some FFI handling outside our code (we just access .ref) but I still plan to look into that after finishing with flatbuffers.

[greenrobot]

I'll post profiler screenshots for state after each commit in #164

Thanks, one after the latest commit should do.

PS.: Does "Dart VM" correspond to Flutter's release mode? This is the most relevant runtime and we want to be sure that we are profiling the right thing.

PPS.: CRUD results per commit may make more sense to see the individual gains (if you want to do that)