Investigate automatic diagnosis of models

[GaetandeCast]

What would you like to say?

Let's explore the different ways we can generate some diagnosis on a model/a ds experiment, using all the information that is computed by the skore reports. This could later be fed to an llm to generate suggestions to the user on how to improve their model/pipeline or avoid a pitfall. The exact UX still needs to be refined but let's start by listing what would be relevant and feasible.

One example of such diagnosis is checking the consistency of metrics values across cross validation splits and warning when the model performs suprisingly good or bad on one specific split. This idea was explored in #1989 and #2318.

[GaetandeCast]

Here is a first set of pitfalls that we could detect. I have separated them in 3 categories:

• Modeling: methodological checks on the modeling part of the data science process. This contains the checks that are the most relevant for skore IMO, and that most would be straightforward to implement

Failure	Description	Typical signal	Feasibility in skore
Overfitting	Model fails to generalize	Large train/test gap in scores and feature importance	Easy
Underfitting	Model fails to capture signal	Low and similar train/test scores, both close to baseline	Easy
Lack of threshold choice	Default decision threshold used despite asymmetric costs	Default threshold (e.g. 0.5), no evidence of tuning vs cost	Easy
Coefficients of unscaled features	Linear coefficients not comparable in magnitude	No scaling on features used in linear model	Easy
Importance of correlated features	Correlated features give unstable importance	Feature correlation near 1 for important features	Easy
MDI of high-cardinality features	Tree MDI inflated for high-cardinality features	Cardinality of feature close to n_samples	Easy
CV variance	Performance unstable across CV folds	High variance or presence of “bad folds” in CV scores	Easy
Improper metric for class imbalance	Accuracy used alone on skewed data	High class imbalance, only accuracy reported	Easy
Degenerate predictions	Model predicts almost only one class or constant values	Very low entropy in predicted labels/probabilities, low prediction variance	Easy
Important Hyperparams untuned	Some hyper parameters (e.g. learning rate) should be tuned for optimal performance	Important hyper parameter left at default value	Easy
constant random state across folds	Constant random state of a model across folds means the random characteristics of the model are fixed	in cross val, the random state of the estimator is an int	Easy
Poor calibration	Probabilities not matching observed frequencies	Bad calibration curve, high Brier score/ECE	Medium
Label noise / inconsistent labels	Contradictory labels for similar samples	Near-duplicate X with different y, high train error for flexible model	Medium
Unstable regularization	Regularization too strong or too weak	Coefs almost all zero with poor perf, or huge coefs with unstable CV	Medium
Metric/optimization mismatch	Objective not aligned with evaluation metric	Loss function differs from metric used for selection	Medium
Model hypothesis violation	Linear model assumptions violated	Nonlinear/funnel pattern in residual plots	Hard (visual/heuristic)
Non interpretable model	Complex model where interpretable one would suffice	Small performance gain vs simpler/interpretable model	Hard (baseline choice)
Large model	Heavy model where small one would suffice	Small performance gain vs lighter model	Hard (baseline choice)

• Pipeline: Checks on common pitfalls related to the pipeline from data to model. Some of these require knowledge of the full pipeline which is not always the case in skore.

Failure	Description	Typical signal	Feasibility in skore
Feature redundancy	Features have been duplicated during preprocessing	feature–feature correlation of 1	Easy
Train-test overlap in time series	Future points in train vs test	Max train timestamp > min test timestamp, shuffled time splits	Easy
Data leakage (general)	Test target info leaked into train	overoptimistic model: cv score >> held out score	Easy
Improper handling of rare categories	Rare/unseen categories mishandled	Many categories mapped to “unknown”, very low-frequency categories	Medium
High-cardinality one-hot	One-hot applied to huge cardinality	Very high cardinality combined with one-hot encoding	Medium
Inconsistent preprocessing train/test	Different transforms between train and test	Mismatched columns/dtypes, many unknown categories in test	Needs full pipeline
Non-serializable steps	Pipeline cannot be safely reused	Custom steps lacking stable params	Needs full pipeline
No random state	Non-reproducible experiments	Stochastic steps without explicit random seed	Needs full pipeline
No scaling of features	Scale sensitive model gets raw features	No scaler before regularized model/SVM/MLP	Needs full pipeline
Data leakage (preprocessing)	Target info leaked via transforms	Scaling/encoding/feature selection fit before split/CV	Needs full pipeline
Target encoding leakage	Target encoders not out-of-fold	Target encoding fitted on full data outside CV	Needs full pipeline

• Data: Checks on the proper handling of the data. These are a bit more outside the scope of skore I think. Although the signals to detect these failures are available using skrub.

Failure	Description	Typical signal
Train-test drift (global)	Data distribution differs between splits	Distance between train/test distributions (global or multivariate)
Per-feature drift	Individual features drift across splits	KS/chi-square tests show large shift in top-k features
Class imbalance	Some classes underrepresented	Class ratio below threshold; some classes rare or missing in train
Label shift	Class priors differ train vs test	Large divergence in class distributions between splits
Wrong data type	Encoded with wrong dtype	Numbers/booleans as strings, categoricals as ints without categories, etc
Mixed-type columns	Dirty/mixed categories	Column mixing numerics and strings, heterogeneous formats
High missing values	Too many missing values to be useful	Column/sample missingness > threshold (e.g. 90%)
Imbalanced missingness patterns	Missingness differs by split/label	Missingness strongly correlated with target or split
Outlier detection	Extreme/unusual values	Points far from bulk (z-score, IQR, isolation methods)
Non IID sampling	Dependence or ordering issues	Strong temporal or grouped structure not accounted for
Circular domain mis-encoding	Circular feature treated linearly	Angles, hours encoded as raw integers/floats without wrap-around
Tokenizable string	Long free-text stored as IDs	Long unique strings with many tokens and high cardinality
Constant/low-variance features	Carry almost no information	One unique value or variance below threshold
Leakage-like features	Almost perfectly predictive single features	Very high univariate AUC/R² for a single feature
Inconsistent units/ranges	Same feature with changed scale	Very different min/max/quantiles between train and test
Target distribution issues (regression)	Skewed or heavy-tailed target	High skewness/kurtosis, many extreme y values
Target distribution issues (classification)	Missing or rare labels	Labels present only in test or only in train
Temporal anomalies	Gaps, duplicates, disorder	Irregular time deltas, duplicates, overlapping time windows
Group leakage	Same entity in train and test	Shared IDs (user, patient, etc.) across splits

[GaetandeCast]

In terms of API, the first idea is to have a diagnose method at different levels of the API:

• at the method/display level. For instance on the coefficients display, the diagnose method alerts in the case of correlated features and warns that if the features are not scaled, the magnitude of coefficients should not be compared
• at the accessor level, the diagnose method runs the diagnosis on of all its methods and presents them in a synthesized way
• same at the report level.

To me, it is important to have this diagnosis be opt-in, not run by default and to output helpful strings, not errors/warnings. This would avoid the friction that is currently in skore.train_test_split.

An unrelated thought is that since the end goal is to feed the signal to an LLM that generates a detailed and personalized report, we should then add more general guidelines to a SKILLS.md type of file that contains a curated set of best practices in ML, such as those found in this and this document.

[GaetandeCast]

As mentioned by @adrinjalali, an other thing we could do is checks on the hyperparameters of the scikit-learn models used like what is done here: https://github.com/SERG-Delft/dslinter/blob/main/dslinter/checkers/hyperparameters_scikitlearn.py
The idea is to warn when some important hyperparameter has not been explicitly set, such as the learning rate in gradient boosting.