Recreating Experiments for "On the Effect of Gradient Regularisation on Interpretability"

This repository contains the code to reproduce the experiments from the paper "On the Effect of Gradient Regularisation on Interpretability".

Installation

Install the dependencies listed in requirements.txt:

pip install -r requirements.txt

Dataset-Model Selection

The dataset-model combination is determined by an input argument --dataset_model. The available options are:

Dataset	Model	Argument Value
MNIST	LeNet	MNIST_LeNet
MNIST	ResNet	MNIST_ResNet
Fashion-MNIST	LeNet	FMNIST_LeNet
KMNIST	LeNet	KMNIST_LeNet
CIFAR-10	ResNet	CIFAR_ResNet
Imagenette	ResNet	Imagenette_ResNet

Directory Structure

All input and output files are stored relative to the path specified with the --data_root argument.

Running the Experiments

1. Model Training

To train models with varying regularisation strengths, run the following command:

python main.py --dataset_model <DATASET_MODEL>

Replace <DATASET_MODEL> with one of the values from the table above.

2. Saliency Evaluation

To evaluate saliency metrics, run:

python saliency.py --dataset_model <DATASET_MODEL> --saliency_method <SALIENCY_METHOD>

Replace <SALIENCY_METHOD> by the method that should be used. Method names are defined by the quantus package: Saliency for the input gradient and DeepLift for DeepLift.

3. Plotting the Results

To generate plots of the metrics specified in USED_METRICS, use:

python plot_saliency.py

For the saliency maps in the appendix, use:

python plot_saliency_maps.py

Replace <DATASET_MODEL> with one of the valid dataset-model options.

4. Generating p-values

To generate the p-values to test whether the faithfulness estimate is actually increasing, the following can be run after the metrics have been calculated:

python p_value.py