The future of fitting frameworks

[ikrommyd]

Especially in CMS, fitting is dominated by a single framework. Combine.
However, analyses like the W mass measurement had to develop their own fitting framework with autodiff because combine just wasn't good enough for them. What does that tell us about the future of fitting frameworks? Do we need a new "roofit" with autodiff and a pythonic interface that is fast? Do we need roofit as is to build an HL-LHC fitting framework around it? Does it need improvements? Does it need a more pythonic API? Do we need smaller fitting frameworks in order to provide speed (one to do just teamplate fits well and fast for example and another for parametric models)?

[guitargeek]

This is a very interesting subject!

I think for more context it would be very nice to get a summary/some pointers from a CMS physicist about the mathematical details of the W mass measurement, in particular to know what exactly were the requirements of the W mass measurement that made Combine unsuitable. I know from private conversations that part of it was definitely the bookkeeping of histograms in ROOT files, which got quite slow for the huge number of template histograms involved in the analysis. But this is not really a limitation of the core of the fitting framework, but how this data is managed. Also, the Combine fork of the W mass measurement was started about a decade ago, and the former limitations are maybe not there anymore.

As the main RooFit developer, of course I advocate for fixing the bottlenecks that arise as we increase the scale of our analyses in Combine, RooFit, or Minuit 2, or Clad (the AD engine of RooFit). But for this, we would need some open data analyses equivalent to the W mass measurement that we can use to optimize and benchmark. The same would also be very helpful for developing a new framework.

Do we need smaller fitting frameworks in order to provide speed (one to do just teamplate fits well and fast for example and another for parametric models)?

Such frameworks would already exist with pyhf and zfit. So I think the same argument as for RooFit applies: what speaks against optimizing these existing packages to make them scale well for analyses like the W mass measurement?

[ikrommyd]

Very valid points @guitargeek. I'd like to point out that I'm not taking any sides here and I'm also not trying to start any fights. This is just how I see the current landscape of fitting frameworks and I'm just raising questions to spark discussion. I edited the issue above to add a couple of more sentences about roofit actually.

[jonas-eschle]

Fully agree with both of you (and for the record: Iason is well aware of pyhf & zfit (one of the authors actually), Jonas is very open to any development anywhere with new tools in general), I think you're on the same line.

Good topic to discuss! I'll probably gonna have a talk as an overview, but it's worth to chat on the different tools! (including what new tools like evermore are able to do).

Including benchmarks that help spot bottle necks, being able to improve the existing tools could help a lot