add MonitoringSource

[TjarkMiener]

This PR prepares ctapipe to interface with camera calibration data products from calibpipe via a new MonitoringSource().

Fixes #2631
Fixes #2565
Fixes #1870
Fixes #1397
Fixes #1164
Fixes #1041
Fixes #856
Fixes #747

[TjarkMiener]

Coverage is low as expected because we do not have a monitoring file running in the CI and proper unit tests.

[maxnoe]

As commented inline, I think this must be implemented independently of any particular event source implementation with a new type of source.

[TjarkMiener]

Thanks for the valuable suggestion, @maxnoe! I wonder if this MonitoringSource can be a child from the HDF5EventSource rather than be an independent component, since we are using the merger component to create the monitoring file. It might just be a matter of overwritting the _generator() in the MonitoringSource child to suit our use case. What do you think?

[maxnoe]

A child? As in the MonitoringSource is an EventSource? No, that's exactly what I want to avoid.

I want to introduce the new concept of something that attaches information by timestamp to an existing event, not creates new events.

This has been discussed a couple of times in issues and no is the right time to implement it:

#1041

[maxnoe]

Minor comment on the docs text. Code-wise this looks fine for me now, but I'd like to ping @FrancaCassol for a review here as the original author of most of the calibration code.

[FrancaCassol]

Hi, the approach of this PR is fine in general. I have just a question, some of the added classes and tools seems to me more specific to calibPipe than to ctapipe, they will be moved to calibPipe later or the idea it to keep most of the code in one library?

[TjarkMiener]

Hi, the approach of this PR is fine in general. I have just a question, some of the added classes and tools seems to me more specific to calibPipe than to ctapipe, they will be moved to calibPipe later or the idea it to keep most of the code in one library?

Hi @FrancaCassol! Thanks for having a look at the PR. The classes and tools that are currently in ctapipe are those that either already serve, or could serve in the future, more general purposes, such as aggregating statistics, detecting outliers, pixel monitoring, chunk interpolation, and so on. calibPipe, on the other hand, is intended to hold only the tools specifically related to calculating the camera calibration coefficients for this use case. Additionally, the interface via MonitoringSource fits naturally in the middleware layer.

[FrancaCassol]

Hi @FrancaCassol! Thanks for having a look at the PR. The classes and tools that are currently in ctapipe are those that either already serve, or could serve in the future, more general purposes, such as aggregating statistics, detecting outliers, pixel monitoring, chunk interpolation, and so on. calibPipe, on the other hand, is intended to hold only the tools specifically related to calculating the camera calibration coefficients for this use case. Additionally, the interface via MonitoringSource fits naturally in the middleware layer.

I see, some of the classes seems to me pretty specific to monitoring issues, but indeed it is never easy to decide where to put interfaces. My only worry is that this development is mainly based on the LST calibration camera concept. For other cameras, things will most probably work differently, so I would have kept the interface at a higher level in order to guarantee independency to further calibPipe developments. That said, I am going to approve this ;-)

[mexanick]

LGTM

[maxnoe]

@kosack ok for me too now, please have a final look and merge if ok

[TjarkMiener]

@kosack ok for me too now, please have a final look and merge if ok

Hi @kosack, when you have a moment this week, could you please take a look at this PR? Your feedback would be really helpful. Let me know if you need anything from my side. It would be good if we can proceed this week with the integrating of the MonitoringSource within data/calibpipe! Many thanks in advance!

[kosack]

Looks very good, other than a few issues and one bug that I found.

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
6 Fixed issues
10 Accepted issues

Measures
0 Security Hotspots
91.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube