f-divergence

$D_f(P||Q) = \int_x q(x)f\left(\frac{p(x)}{q(x)}\right)dx$

且f(x) is convex && f(1) = 0

$D_f(P||Q)$用于评估分布P和分布Q的差异。
f(1) = 0 ==> 当p(x)=q(x)时，Df = 0
f is convex ==> Df >= f(1) = 0

令$f(x) = x\log x$，则Df是KL divergence
$D_f = \int_x p(x)\log\left(\frac{p(x)}{q(x)}\right)$

令$f(x) = -\log x$，则Df是reverse KL divergence
$D_f = \int_x q(x)\log\left(\frac{q(x)}{p(x)}\right)$

令$f(x) = (x-1)^2$，则Df是Chi Square Divergence
$D_f = \int_x\frac{(p(x)-q(x))^2}{q(x)}dx$

KL散度 KL divergence

有两个独立的分布P(x)和Q(x)，KL散度用于衡量这两个分布的差异。

$D_{KL}(P||Q) = E_{X\sim P}\left[\log \frac{P(x)}{Q(x)}\right] = E_{X\sim P}[\log P(x) - \log Q(x)]$

$H(P, Q) = -E_{X\sim P} \log Q(x)$

最小化交叉熵等价于最小化KL散度。

http://blog.sina.com.cn/s/blog_18bdda0da0102xzpw.html