KL Divergence Between Two PMFs

Computes the Kullback-Leibler divergence \(D_{KL}(p \| q)\) between two probability mass functions.

Usage

kl_divergence_pmf(p, q, eps = NULL)

Arguments

p

Numeric vector; target PMF (reference distribution).

q

Numeric vector; comparison PMF.

eps

Deprecated compatibility argument. Exact mathematical KL does not smooth zero probabilities; supplying a non-NULL value is an error.

Value

Numeric scalar; the KL divergence (non-negative).

Details

The KL divergence is defined as: $$D_{KL}(p \| q) = \sum_k p(k) \log\frac{p(k)}{q(k)}$$

Terms with \(p(k)=0\) contribute zero. If \(p(k)>0\) and \(q(k)=0\), the result is Inf. Both inputs must already be valid, normalized PMFs; this function never normalizes or smooths them. In keeping with the package PMF contract, a sum differing from one by at most .TOL_PMF_SUM is accepted without modifying the vector. A raw negative value whose magnitude is fully explained by that validation tolerance and floating-point roundoff is reported as zero.

Properties:

• \(D_{KL}(p \| q) \geq 0\); for exactly normalized PMFs, equality holds iff \(p = q\). For sums accepted within the PMF tolerance, equality is interpreted up to that same numerical tolerance.

• Not symmetric: \(D_{KL}(p \| q) \neq D_{KL}(q \| p)\)

See also

kl_divergence_K for KL divergence with induced PMF

Examples

p <- c(0.2, 0.5, 0.3)
kl_divergence_pmf(p, p)  # 0
#> [1] 0

q <- c(0.3, 0.4, 0.3)
kl_divergence_pmf(p, q)
#> [1] 0.03047875