KL Divergence Between Two PMFs

Computes the Kullback-Leibler divergence \(D_{KL}(p \| q)\) between two probability mass functions.

Usage

kl_divergence_pmf(p, q, eps = 1e-15)

Arguments

p

Numeric vector; target PMF (reference distribution).

q

Numeric vector; comparison PMF.

eps

Numeric; small value to prevent log(0). Default: 1e-15.

Value

Numeric scalar; the KL divergence (non-negative).

Details

The KL divergence is defined as: $$D_{KL}(p \| q) = \sum_k p(k) \log\frac{p(k)}{q(k)}$$

Only indices where \(p(k) > \epsilon\) are included in the sum.

Properties:

• \(D_{KL}(p \| q) \geq 0\) with equality iff \(p = q\)

• Not symmetric: \(D_{KL}(p \| q) \neq D_{KL}(q \| p)\)

See also

kl_divergence_K for KL divergence with induced PMF

Examples

p <- c(0.2, 0.5, 0.3)
kl_divergence_pmf(p, p)  # 0
#> [1] 0

q <- c(0.3, 0.4, 0.3)
kl_divergence_pmf(p, q)
#> [1] 0.03047875