Chunked model application

[maxnoe]

No description provided.

[codecov]

Codecov Report

Base: 92.74% // Head: 92.74% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (b427dbd) compared to base (8004264).
Patch coverage: 95.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2133      +/-   ##
==========================================
- Coverage   92.74%   92.74%   -0.01%     
==========================================
  Files         214      214              
  Lines       17914    17941      +27     
==========================================
+ Hits        16615    16639      +24     
- Misses       1299     1302       +3     

Impacted Files	Coverage Δ
ctapipe/io/metadata.py	96.58% <ø> (ø)
ctapipe/tools/tests/test_process_ml.py	100.00% <ø> (ø)
ctapipe/tools/apply_models.py	93.10% <87.50%> (-3.15%)	⬇️
ctapipe/io/tableloader.py	97.58% <96.42%> (+0.15%)	⬆️
ctapipe/conftest.py	94.58% <100.00%> (+0.31%)	⬆️
ctapipe/io/tests/test_table_loader.py	99.47% <100.00%> (+0.02%)	⬆️
ctapipe/tools/tests/test_apply_models.py	100.00% <100.00%> (ø)
ctapipe/tools/tests/test_train.py	100.00% <100.00%> (ø)
ctapipe/visualization/mpl_array.py	99.24% <0.00%> (-0.01%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[maxnoe]

This is successful in that it reduced memory consumption a lot, but it's also a lot slower and I think it doesn't make sense to merge until we remove the bottlenecks from the table loader for that. I will try to incorporate this.

[maxnoe]

Ok, main problem was that adding the telescope trigger information naively was very slow, since it required loading the whole trigger telescope for each chunk.

Now, application is much faster (even a lot faster than non-chunked, since joining N chunks is faster than joining the whole table).

[kosack]

Looks good! The effect on memory useage is clear. I tried here some extreme cases: basically using 100k events (blue) vs 1k events (black):

Of course the optimal for speed is clearly at the higher end, but it's nice we can control the memory usage so well