# Variational Autoencoders = Minimizing Prior Distribution + Maximizing Mutual Information

Source: <https://kexue.fm/archives/6088>

Machine translated. Refer to the original post to verify accuracy.

---
author: Jianlin Su
date: October 10, 2018
lang: en
markdown_url: 06088-Variational-Autoencoder-Minimizing-Prior-Distribution-Maximizing-Mutual-Information.md
next_url: 06051-Lipschitz-Constraints-in-Deep-Learning-Generalization-and-Generative-Models.html
post_id: 6088
prev_url: 06096-Rethinking-the-Determinant-of-Non-Square-Matrices.html
site_title: English (unofficial) translations of posts at kexue.fm
source_url: "https://kexue.fm/archives/6088"
title: Variational Autoencoders = Minimizing Prior Distribution + Maximizing Mutual Information
---

This article is very short, mainly describing a very useful, not complicated, but surprisingly long-overlooked fact.

In the article [Mutual Information in Deep Learning: Unsupervised Feature Extraction](./06024-Mutual-Information-in-Deep-Learning-Unsupervised-Feature-Extraction.html), we obtained the final loss for the Deep INFOMAX model through a weighted combination of a prior distribution loss and a maximized mutual information loss. While that story was complete, in a sense, it was just a cobbled-together loss. This article aims to prove that the loss can be naturally derived from Variational Autoencoders (VAE).

# Process

To repeat once more, the loss that the Variational Autoencoder (VAE) needs to optimize is:
``` math
\begin{equation}
\begin{aligned}
&KL(\tilde{p}(x)p(z|x)\Vert q(z)q(x|z))\\
=&\iint \tilde{p}(x)p(z|x)\log \frac{\tilde{p}(x)p(z|x)}{q(x|z)q(z)} dzdx
\end{aligned}
\end{equation}
```
Related discussions have appeared many times on this blog. VAE includes both an encoder and a decoder. If we only need to encode features, training a decoder seems redundant. Therefore, the focus is on how to remove the decoder.

It’s actually quite simple: split the VAE loss into two parts:
``` math
\begin{equation}
\begin{aligned}
&KL(\tilde{p}(x)p(z|x)\Vert q(z)q(x|z))\\
=&\iint \tilde{p}(x)p(z|x)\log \frac{p(z|x)}{q(z)} dzdx-\iint \tilde{p}(x)p(z|x)\log \frac{q(x|z)}{\tilde{p}(x)} dzdx
\end{aligned}
\end{equation}
```
The first term is the KL divergence of the prior distribution, and isn’t the $`\log \frac{q(x|z)}{\tilde{p}(x)}`$ in the second term actually the pointwise mutual information of $`x`$ and $`z`$? If $`q(x|z)`$ has infinite fitting capacity, eventually we will inevitably have $`\tilde{p}(x)p(z|x) = q(x|z)p(z)`$ (Bayes’ rule), so the second term is:
``` math
\begin{equation}
KL(q(x|z)p(z)\Vert \tilde{p}(x)p(z))=KL(\tilde{p}(x)p(z|x)\Vert \tilde{p}(x)p(z))
\end{equation}
```
which is the mutual information between the two random variables $`x`$ and $`z`$. The negative sign in front means we want to maximize the mutual information.

The rest of the process is the same as in [Mutual Information in Deep Learning: Unsupervised Feature Extraction](./06024-Mutual-Information-in-Deep-Learning-Unsupervised-Feature-Extraction.html), and is omitted here.

# Conclusion

As stated at the beginning, this article is very short and concise. Its main purpose is to provide a new understanding of the Variational Autoencoder loss (minimizing prior distribution + maximizing mutual information), which then naturally leads to the loss for Deep INFOMAX.

If I hadn’t already written [Mutual Information in Deep Learning: Unsupervised Feature Extraction](./06024-Mutual-Information-in-Deep-Learning-Unsupervised-Feature-Extraction.html), I would have definitely used this starting point to explain Deep INFOMAX. However, since that article has been out for a few days, I had to write this short post as a supplement.

<span style="color: FF8800">***When reposting, please include the address of this article:** [https://kexue.fm/archives/6088](./06088-Variational-Autoencoder-Minimizing-Prior-Distribution-Maximizing-Mutual-Information.html)*</span>

<span style="color: FF8800">***For more detailed reposting matters, please refer to:** [Scientific Space FAQ](./06508-Scientific-Space-Browsing-Guide-FAQ.html)*</span>