Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models

René Haas, Inbar Huberman-Spiegelglas, Rotem Mulayoff, Stella Graßhof, Sami Sebastian Brandt, Tomer Michaeli

Publikation: Konference artikel i Proceeding eller bog/rapport kapitelKonferencebidrag i proceedingsForskningpeer review

Abstract

Denoising Diffusion Models (DDMs) have emerged as a strong competitor to Generative Adversarial Networks (GANs). However, despite their widespread use in image synthesis and editing applications, their latent space is still not as well understood. Recently, a semantic latent space for DDMs, coined ‘h-space’, was shown to facilitate semantic image editing in a way reminiscent of GANs. The h-space is comprised of the bottleneck activations in the DDM’s denoiser across all timesteps of the diffusion process. In this paper, we explore the properties of h-space and propose several novel methods for finding meaningful semantic directions within it. We start by studying unsupervised methods for revealing interpretable semantic directions in pretrained DDMs. Specifically, we show that interpretable directions emerge as the principal components in the latent space. Additionally, we provide a novel method for discovering image-specific semantic directions by spectral analysis of the Jacobian of the denoiser w.r.t. the latent code. Next, we extend the analysis by finding directions in a supervised fashion in unconditional DDMs. We demonstrate how such directions can be found by annotating generated samples with a domain-specific attribute classifier. We further show how to semantically disentangle the found directions by simple linear projection. Our approaches are applicable without requiring any architectural modifications, text-based guidance, CLIP-based optimization, or model fine-tuning.
OriginalsprogEngelsk
Titel18th IEEE International Conference on Automatic Face and Gesture Recognition
Publikationsdato2024
StatusUdgivet - 2024
BegivenhedIEEE International Conference on Automatic Face and Gesture Recognition - Istanbul, Tyrkiet
Varighed: 27 maj 202431 maj 2024
Konferencens nummer: 18
https://fg2024.ieee-biometrics.org/

Konference

KonferenceIEEE International Conference on Automatic Face and Gesture Recognition
Nummer18
Land/OmrådeTyrkiet
ByIstanbul
Periode27/05/202431/05/2024
Internetadresse

Fingeraftryk

Dyk ned i forskningsemnerne om 'Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models'. Sammen danner de et unikt fingeraftryk.

Citationsformater