This is the official implementation of the decoder introduced in the paper "Feature Space Analysis by Guided Diffusion Model", which will be presented at ACCV 2026.
This paper aims to analyse the feature space of a vision-related Deep Neural Network (DNN) by proposing a decoder that can generate an image whose feature closely matches a user-specified feature. Supported by quantitative evidence of this close feature matching, our decoder facilitates precise analysis of the DNN's feature space. Our decoder is implemented as a guided diffusion model that guides the image generation of a pre-trained diffusion model to minimise the Euclidean distance between the feature of a clean image estimated at each step and the user-specified feature. The key advantages of our decoder are its training-free applicability to analyse the feature spaces of different DNNs and its practical feasibility on a single COTS GPU. The experiments targeting CLIP's image encoder and ResNet-50 demonstrate the effectiveness of our decoder both as a feature-matching image generator and as a visual feature space analyser.
Our decoder has been implemented by modifying a universal guided diffusion model developed by A. Bansal et al. and runs in the same environment as it. This environment can be constructed by the following commands:
conda env create -f environment.yaml
conda activate ldm
conda install pytorch torchvision cudatoolkit=11.3 -c pytorch
pip install GPUtil
pip install blobfile
pip install facenet-pytorch # This is necessary for pytorch's compatibility with CUDA
pip install --upgrade transformers huggingface_hub requests
To ease the reproduction of our decoder's results, we release Dockerfile to create the same environment as ours in which our decoder is implemented and tested on Ubuntu 22.04 and CUDA 11.7.1. The other libraries we used can also be seen in this Dockerfile. Furthermore, our anaconda environment was created by this installer.
The stable diffusion (sd-v1-4.ckpt) that is used as a background diffusion model of our decoder can be found here and should be placed in models/ldm/.
-
--optim_forward_guidance: Flag to use the guidance to generate an image whose feature minimises its squared Euclidean distance to a user-specified one -
--optim_forward_guidance_wt: Weight for the graident of the loss fuction ($w_g$ in the paper) -
--optim_num_steps: Number of self-recurrence iterations ($K$ in the paper, multiple numbers can be specified) -
--optim_early_emp_end: Index of the last step for which the early step emphasis is used ($T - T'$ in the paper) -
--optim_grad_clip_threshold: Threshold for gradient clipping ($\nabla_{thres}$ in the paper) -
--optim_folder: Name of the output directory where the generated image as well as images at intermediate steps and the log file are stored. The finally generated image and the best image are saved intoout_img.pngandout_img_best_t_<step #>_rec_<self rec. #>_bk_-1_sim_-<squared Euclidean distance>.png, respectively. -
--prompt: Used to specify an image (png or jpg) or a text prompt from which the user-specified feature is extrated
Please check scripts/clip_guided.py, scripts/resnet50_guided.py or scripts/vith14_guided.py for more details about variables.
In addition, when you want to analyse the feature space of another feature extractor, what you need to do is only to copy one of these script files and change the class handling the feature extraction process (i.e., Clip in scripts/clip_guided.py, Resnet50 in scripts/resnet50_guided.py and VitH14 in scripts/vith14_guided.py).
Using CLIP's image encoder with ResNet-50 backbone as the target feature extractor, the following command is used to generate an image whose feature closely matches the feature of an actual image (e.g., data/bansal/og_img_0.png below):
python scripts/clip_guided.py --optim_forward_guidance --optim_forward_guidance_wt 4 --optim_num_steps 1000 8 --optim_early_emp_end 995 --optim_grad_clip_threshold 3.0 --optim_folder data/bansal/og_img_0.png.generated_check --optim_print --prompt data/bansal/og_img_0.png
The following command is used to try to generate an image whose feature closely matches the feature of a textual caption (e.g., "A black motorbike parked on the dry ground" below):
python scripts/clip_guided2.py --optim_forward_guidance --optim_forward_guidance_wt 4 --optim_num_steps 1000 8 --optim_early_emp_end 995 --optim_grad_clip_threshold 3.0 --optim_folder data/mscoco_data/txt_1.generated --optim_print --prompt "a black motorbike parked on the dry ground"
Using ResNet-50, especially ResNet-50 configured with ResNet50_Weights.IMAGENET1K_V1, as the target feature extractor, the following command is used to generate an image whose feature closely matches the feature of an actual image (e.g., data/bansal/og_img_0.png below):
python scripts/resnet50_guided.py --optim_forward_guidance --optim_forward_guidance_wt 4 --optim_num_steps 1000 8 --optim_early_emp_end 995 --optim_grad_clip_threshold 3.0 --optim_folder data/bansal/og_img_0.png.generated_check_res --optim_print --prompt data/bansal/og_img_0.png
Using vision transformer, especially ViT-H/14 configured with ViT_H_14_ Weights.IMAGENET1K_SWAG_E2E_V1, as the target feature extractor, the following command is used to generate an image whose feature closely matches the feature of an actual image (e.g., data/bansal/og_img_0.png below):
python scripts/vith14_guided.py --optim_forward_guidance --optim_forward_guidance_wt 4 --optim_num_steps 1000 8 --optim_early_emp_end 995 --optim_grad_clip_threshold 2.0 --optim_folder data/bansal/og_img_0.png.generated_check_res --optim_print --prompt data/bansal/og_img_0.png
If you found our decoder useful, please cite the following paper:
@inproceedings{shirahama2026fsa,
author={Shirahama, Kimiaki and Yamashita, Kaduki and Yanobu, Miki and Ohsaki, Miho},
title={Feature Space Analysis by Guided Diffusion Model},
booktitle={Proc. of the 18th Asian Conference on Computer Vision (ACCV 2026)},
year={2026},
note={To appear}
}
