Post-Sprint actions

[bthirion]

• Make classes for Knockoffs, DL, CluDL, EnCluDL (one PR for each, including tests and examples)
• Then implement ALE, PDP, SAGE with the same type of API.

[lionelkusch]

This is an example of the new API with class: #220

CPI, PFI, LOCO don't have a function in their actual implementation and the score method doesn't respect the new API.

[lionelkusch]

We need 2 add at least 2 examples:

• on the issue of Permutation Feature Importance (see Add classification example on the iris dataset #229 )
• on the difference between marginal and conditional importance (see Conditional vs marginal importance on the XOR dataset #218)

[bthirion]

• For the PFI issue I think the californian housing does it, right ?
• For the second, i agree that a basic example explaining how to interpret marginal vs conditional importance is welcome

[lionelkusch]

During the sprint of the 4/4/2025, we define 4 types of methods:

VI: variable importance
Methods which provide only an estimation of importance by variables and don't have statistical control of their results.
For example: PDP(Partial Dependence Plot), ALE (accumulated local effect), PFI*(permutation Feature Importance), SAGE*(Shapley Additive Global importancE)
Minimal Input: X, Y, alpha (threshold for selection)
Additional Input:

• PFI: estimator
• SAGE: estimator, conditional sampler
• PDP: precision of the curve
• ALE:

Output: selection, importance
Additional Output:

• PFI:
• SAGE: estimator
• PDP: ICE (Individual Conditional Expectation)
• ALE: ICE, M-Plots (marginal plot)

CVI: control variable importance
Methods which provide an estimation of importance by variables and a confidence interval/p-value based on it, we can have a selection of important variables.
For example: CPI*(Conditional Permutation Importance), LOCO*(Leave One Covariate Out), Desparsified Lasso, Marginal/Correlation?
Minimal Input: X, Y, alpha (threshold for selection)
Additional Input:

• LOCO: estimator
• CPI: estimator, conditional sampler
• DL: Lasso parameters
• Marginal:

Output: selection, importance
Additional Output:

• LOCO: estimators
• CPI: estimator, conditional samplers
• DL: 3xLasso
• Marginal: correlation matrix

IVS: individual variable selection
Methods which provide a p-value of each variable about their importance. Based on this p-value, we can select some important variables.
For example: D0CRT(distillation 0 conditional residual test), HRT(holdout randomisation test), GCM(generalised covariance measure), CDL(Cluster Despasified Lasso)
Minimal Input: X, Y, alpha (threshold for selection)
Additional Input:

• D0CRT: method of y-distillation
• CDL: parameters for DL, feature aggregator

Output: selection, p-value
Additional Output:

• D0CRT: residual of x, sigma X, Lassos, residual of y, model for distillation y
• CDL: DLs

MVS: multiple variable selection
Methods which selection variables are based on an FDR (False Discovery Rate) or FDP (False Discovery Proportion) quantity and a score.
For example: KO (KnockOff), AKO* (Aggregation KnockOff), EnCluDL (ensemble clustering Desparsified Lasso)
Minimal Input: X, Y, alpha (FDR/FDP)
Additional Input:

• KO: estimator distribution
• ENCluDL: parameters of CDL and parameters for bootstrapping

Output: selection, score
Additional Output:

• KO: threshold, estimated distribution, Lasso, LDS/Kostatistic
• ENCluDL: CDLs

For getting confidence intervals, it's possible to use the generic methods:

• HULC (Hull based Confidence)
• BootStraping

*: support group of variables
?: possibility of support group of variable

---

The method will be written in class and a function associate to them will be added to simplify the interaction with the data.
Each will have 4 methods:

• __init__(parameters): for setup the parameters.
• fit(X, y, parameters connected to the data): for fitting the method to some data
• importance(): for returning the importance or selection of the method
• selection (f_threshold): return the selection of variable base on threshold

---

This is the definition of the Conditional sampler:
Condional_sampler():

• __init__(parameters)
• fit(X)
• sample(condition_permutation)

[lionelkusch]

@bthirion @jpaillard @AngelReyero @alexblnn
I try to summarise here the conclusion of your discussion during the spring.
If I miss something or there require some correction, don't hesitate to mention it.
Following the discussion, we should open some issues for going more into detail.

[AngelReyero]

Thanks, I have some comments:

• For SAGE, as additional input there is conditional sampler and estimator. What do you mean by conditional sampler? Also, for estimator, it is just some general estimator that will be fitted for each subset? Then, as output, is it all the fitted estimators?
• For the CVI, I think it is better controlled variable importance. Also, for the IVS and MVS, maybe mention that it is conditional variable selection? Or keep it general to have also marginal independence methods such as marginal correlation?
• For the confidence intervals, particularly for the LOCO and CPI, we should add the correctional square root term in order to get it valid (see Verdinelli & Wasserman 2023). Also, for the LOCO, maybe it could be interesting to have the correction of Williamson et al. 2021.
• Finally, I think that it could be interesting to add an example to illustrate the difference between Variable importance and variable selection, i.e. that having an heuristic importance close to 0 does not provide any insights of the conditional null hypothesis for the variable. A good example is given in the HRT article of Tansey et al. 2021 (Table 3). With real data (a drug response to cancer cell dataset and an olfactory perception dataset), they show some covariates with small variable importance but small p-value, and the other opposite.

[jpaillard]

As discussed, for importance methods that measure a loss difference on a test set, the "fit à la sklearn" is natural. However, I think it can be misleading for other methods (e.g., knockoffs) that do not require a train/test split.
A first suggestion would be not to implement fit for such methods but just selection, for instance.

This remark does not apply to CondionalSampler, for which I think fit makes sense.

[bthirion]

You mean: not implement fit for knockoffs ?
I tend to disagree with that. We could expect fit to compute the knockoffs statistics, that are then used for inference during the selection call?

[jpaillard]

I agree that we need some form of fit in knockoffs as well. My suggestion was to keep it as an internal function rather than exposing the fit.
I fear that users could be misled into using something like:

X_train, X_test = train_test_split(X)
knockoffs.fit(X_train)
knockoffs.selection(X_test)

While this method doesn't require data splitting.

This point is actually a question, I don't have a strong opinion.

[bthirion]

This is true, and for this reason we need to be explicit in the docstrings, examples and user documentation.

[lionelkusch]

I have two questions linked to this topic.
From my summary, I extracted 4 methods

The method will be written in class and a function associate to them will be added to simplify the interaction with the data. Each will have 4 methods:

* __init__(parameters): for setup the parameters.

* fit(X, y, parameters connected to the data): for fitting the method to some data

* importance(): for returning the importance or selection of the method

* selection (f_threshold): return the selection of variable base on threshold

I don't remember that the method importance will have a parameter X.
Which methods or type of methods should have a parameter X for the importance?

By refactoring D0CRT, I realise that the output of importance is only the p-values. Do we also need a selection variable of importance at the same time or the method selection is enough?