Information Geometry & Fisher Metric

Overview & Motivation

The space of probability distributions is not just a set — it is a Riemannian manifold with a canonical metric. This observation, due independently to Rao (1945) and later systematized by Amari (1985), places statistical inference squarely inside the Riemannian framework we built in Riemannian Geometry and Geodesics & Curvature.

Here is the key intuition. Consider two pairs of Gaussians:

• $\mathcal{N}(0, 1)$ and $\mathcal{N}(0.01, 1)$ — the means differ by $0.01$, variance is $1$.
• $\mathcal{N}(0, 0.01)$ and $\mathcal{N}(0.01, 0.01)$ — the means differ by the same $0.01$, but the variance is $0.01$.

In Euclidean parameter space, these two pairs are the same distance apart: $|\Delta\mu| = 0.01$. But statistically, the second pair is far more distinguishable — a variance of $0.01$ means the distributions are tightly concentrated, so a shift of $0.01$ in the mean is enormous relative to the spread. The Fisher information metric captures this: the “true” distance between distributions depends on where you are in parameter space, exactly as a Riemannian metric varies from point to point on a manifold.

This topic brings together the three pillars of the Differential Geometry track. The parameter space of a statistical model is a smooth manifold. The Fisher information defines a Riemannian metric on this manifold. And the resulting geodesics and curvature have direct statistical meaning: geodesics are the paths of natural gradient descent, and curvature controls the precision of statistical estimation via the Cramér–Rao bound.

What We Cover

1. Statistical manifolds — parametric families as smooth manifolds, identifiability, and Čencov’s uniqueness theorem
2. The Fisher information metric — score functions, the Fisher matrix, and why it is a Riemannian metric
3. Classical families — the Gaussian manifold as the Poincaré half-plane, Bernoulli geometry, and exponential families
4. $\alpha$-connections — Amari’s one-parameter family, $e$/$m$-duality, and dually flat manifolds
5. Divergence functions — KL divergence, $\alpha$-divergences, Bregman divergences, and the generalized Pythagorean theorem
6. Geodesics — Fisher-Rao geodesics, the Mahalanobis distance, and hyperbolic geometry of variance
7. The Cramér–Rao bound — curvature and estimation precision, efficient estimators
8. Computational notes — symbolic Fisher metric, Christoffel symbols, geodesic solvers, natural gradient
9. Information geometry in ML — natural gradient descent, variational inference, Adam, and optimal transport

Prerequisites

This topic builds directly on all three preceding topics in the Differential Geometry track:

• Smooth Manifolds — charts, tangent spaces, and the differential structure that parameter spaces inherit
• Riemannian Geometry — metric tensors, the Levi-Civita connection, parallel transport, and the machinery for measuring lengths and angles
• Geodesics & Curvature — the geodesic equation, curvature tensors, and the Gauss–Bonnet theorem

We also draw on the Spectral Theorem for eigendecomposition of the Fisher matrix, and connect to PCA & Low-Rank Approximation through the lens of preconditioning.

---

Statistical Manifolds & Parametric Families

The identifiability requirement (condition 3) ensures that the parameter space faithfully represents the set of distributions — there is no redundancy. Without identifiability, the Fisher metric degenerates: it becomes only positive semi-definite rather than positive definite, because some parameter directions produce no change in the distribution.

Examples. The Gaussian family $\mathcal{N}(\mu, \sigma^2)$ has parameter space $\Theta = \mathbb{R} \times \mathbb{R}_+$, a 2-dimensional manifold. The Bernoulli family $\text{Ber}(p)$ has $\Theta = (0, 1)$, a 1-dimensional manifold. The exponential family $\text{Exp}(\lambda)$ has $\Theta = \mathbb{R}_+$. The multinomial on $k$ categories, $\text{Mult}(p_1, \ldots, p_k)$ with $\sum p_i = 1$, has $\Theta$ equal to the open $(k-1)$-simplex.

Each point $\theta \in \Theta$ represents an entire probability distribution $p_\theta$. As we move through the parameter space, we trace out a path through the space of distributions. The tangent space at $\theta$ consists of directions in which we can perturb the parameter — and, as we will see, these tangent vectors can be identified with score functions.

---

The Fisher Information Metric

With the smooth manifold structure in place, we now equip the parameter space with a Riemannian metric. The construction proceeds in three steps: define the score function, take its covariance, and verify that the result is a valid Riemannian metric.

Since the score has zero mean, its covariance matrix is simply $\mathbb{E}[s_i \, s_j]$. This covariance is the Fisher information matrix.

The equivalence of the two forms is a standard computation: differentiate the zero-mean identity $\mathbb{E}[s_i] = 0$ with respect to $\theta^j$ and use the product rule.

Family:Gaussian N(μ, σ²)Bernoulli Ber(p)Exponential Exp(λ)Show metric ellipsesShow score functions

Drag the point to explore how the Fisher metric varies across parameter space. Metric ellipses show the local unit ball in the Fisher metric — smaller ellipses mean higher information.

---

Fisher Metric for Classical Families

We now compute the Fisher metric explicitly for three canonical parametric families. Each reveals different geometric structure.

This is computed by applying the Riemann curvature tensor formula from Geodesics & Curvature to the Fisher metric $g = \text{diag}(1/\sigma^2, 2/\sigma^2)$. The Gaussian manifold is a surface of constant negative curvature — a hyperbolic space. This means that the space of Gaussian distributions, equipped with the Fisher metric, has the same local geometry as the Poincaré half-plane.

This is a striking result: for exponential families, the Fisher metric is simply the Hessian of a single scalar function $A(\eta)$. The convexity of $A(\eta)$ (a standard property of log-partition functions) guarantees positive definiteness — the Fisher metric is automatically a valid Riemannian metric.

---

$\alpha$-Connections and Dual Geometry

The Levi-Civita connection from Riemannian Geometry is the unique torsion-free, metric-compatible connection. Amari’s key insight (1985) is that statistical manifolds carry not one but a one-parameter family of connections — the $\alpha$-connections — and the interplay between them reveals the deepest geometric structure.

The $\alpha$-connections differ from the Levi-Civita connection by a cubic tensor (the skewness tensor or Amari-Chentsov tensor) that vanishes when $\alpha = 0$. The crucial property is duality.

The most important instance of duality is between the $e$-connection ($\alpha = 1$) and the $m$-connection ($\alpha = -1$). This duality underlies the entire structure of exponential families.

For the Gaussian family $\mathcal{N}(\mu, \sigma^2)$, the natural parameters are $\eta_1 = \mu/\sigma^2$ and $\eta_2 = -1/(2\sigma^2)$, and the expectation parameters are $\mu_1 = \mathbb{E}[X] = \mu$ and $\mu_2 = \mathbb{E}[X^2] = \mu^2 + \sigma^2$. Straight lines in $(\eta_1, \eta_2)$-space are $e$-geodesics; straight lines in $(\mu_1, \mu_2)$-space are $m$-geodesics. These are generically different curves.

e-geodesic (α = 1)m-geodesic (α = −1)Levi-Civita (α = 0)Grid:Noneθ (natural params)η (expectation params)

Drag the two endpoints to compare geodesics under different connections. The e-geodesic is straight in natural parameters; the m-geodesic is straight in expectation parameters; the Levi-Civita geodesic (α = 0) follows the Fisher-Rao metric.

---

Divergence Functions

Divergence functions measure the “distance” between probability distributions, but they are not true distances — they violate symmetry and/or the triangle inequality. Nevertheless, they encode the geometry of statistical manifolds and are fundamental to inference and learning.

Despite not being a distance, KL divergence has a deep connection to the Fisher metric.

The generalized Pythagorean theorem connects divergences to the dual geometry of $\alpha$-connections.

This theorem is the information-geometric foundation of variational inference: minimizing $D_{\mathrm{KL}}(q \,\|\, p^*)$ over a variational family is an $m$-projection, and the Pythagorean theorem guarantees that the optimal $q$ decomposes the divergence into an “explained” part and an irreducible “approximation error.”

Divergence:KL(p‖q)KL(q‖p)HellingerFisher-Rao

Drag the two points to compare divergences between Gaussians. KL divergence is asymmetric — swapping p and q changes the value. The Fisher-Rao distance is a true metric (symmetric, triangle inequality).

---

Geodesics on Statistical Manifolds

The Fisher information metric defines geodesics on statistical manifolds via the geodesic equation from Geodesics & Curvature. These geodesics have concrete statistical interpretations and differ dramatically from Euclidean straight lines.

For the Gaussian family, the Fisher metric $ds^2 = \frac{1}{\sigma^2}(d\mu^2 + 2 \, d\sigma^2)$ is (up to a constant) the Poincaré upper half-plane metric. The geodesics are:

• Vertical lines: $\mu = \text{const}$, $\sigma$ varies — changing the variance while keeping the mean fixed.
• Semicircles centered on the $\mu$-axis — the shortest paths between distributions with different means and variances.

These are precisely the geodesics of hyperbolic geometry, consistent with the constant curvature $K = -1/2$.

The Fisher-Rao distance naturally produces the Mahalanobis distance — the standard “number of standard deviations” between means. This is not a coincidence: the Fisher metric is the infinitesimal Mahalanobis metric.

The logarithmic scaling means that doubling $\sigma$ from $1$ to $2$ is the same Fisher-Rao distance as doubling from $100$ to $200$. This is the natural scale for variance: what matters is the ratio, not the absolute difference.

Geodesic fanKL contoursNatural gradient mode

Drag the start point to explore Fisher-Rao geodesics on the Gaussian manifold. These are semicircles in the Poincaré half-plane model.

---

The Cramér–Rao Bound

The Cramér–Rao bound connects the Fisher information metric to the fundamental limits of statistical estimation. It is the information-geometric version of the uncertainty principle: curvature (Fisher information) controls precision (estimator variance).

---

Computational Notes

The computations of information geometry can be automated symbolically and solved numerically.

Symbolic Fisher Metric Derivation

Using SymPy, we can derive the Fisher metric for the Gaussian family from scratch:

import sympy as sp

x, mu, sigma = sp.symbols('x mu sigma', real=True)
sigma = sp.Symbol('sigma', positive=True)

# Log-likelihood
log_p = -sp.log(sigma) - (x - mu)**2 / (2 * sigma**2) - sp.log(sp.sqrt(2 * sp.pi))

# Score functions
s_mu = sp.diff(log_p, mu)          # (x - mu) / sigma^2
s_sigma = sp.diff(log_p, sigma)    # -1/sigma + (x-mu)^2 / sigma^3

# Fisher matrix entries via E[s_i * s_j]
# Using E[(x-mu)^2] = sigma^2, E[(x-mu)^4] = 3*sigma^4
g_11 = sp.Rational(1, 1) / sigma**2
g_22 = sp.Rational(2, 1) / sigma**2
g_12 = sp.Integer(0)
print(f"Fisher metric: diag({g_11}, {g_22})")
# Output: Fisher metric: diag(sigma**(-2), 2/sigma**2)

Christoffel Symbols for the Gaussian Manifold

With coordinates $(\mu, \sigma)$ and metric $g = \text{diag}(1/\sigma^2, 2/\sigma^2)$:

# Christoffel symbols: Gamma^k_{ij} = (1/2) g^{kl}(d_i g_{jl} + d_j g_{il} - d_l g_{ij})
# For diagonal metric g = diag(f, h) where f = 1/sigma^2, h = 2/sigma^2:
# Nonzero symbols:
#   Gamma^sigma_{mu,mu}    = -f'/(2h)   = (2/sigma^3) / (2 * 2/sigma^2) = 1/(2*sigma)
#   Gamma^mu_{mu,sigma}    = f'/(2f)    = (-2/sigma^3) / (2/sigma^2) = -1/sigma
#   Gamma^sigma_{sigma,sigma} = h'/(2h) = (-4/sigma^3) / (2 * 2/sigma^2) = -1/sigma

Numerical Geodesic Solver

The geodesic equation on the Gaussian manifold is the ODE system:

$\ddot{\mu} + 2\,\Gamma^\mu_{\mu\sigma}\,\dot{\mu}\dot{\sigma} = 0, \qquad \ddot{\sigma} + \Gamma^\sigma_{\mu\mu}\,\dot{\mu}^2 + \Gamma^\sigma_{\sigma\sigma}\,\dot{\sigma}^2 = 0$

We solve this with a 4th-order Runge-Kutta integrator:

import numpy as np

def geodesic_step_gaussian(state, dt):
    """Single RK4 step for the geodesic ODE on the Gaussian manifold."""
    mu, sigma, dmu, dsigma = state

    def derivs(s):
        mu, sig, dm, ds = s
        ddmu = (2 / sig) * dm * ds        # -2 * Gamma^mu_{mu,sigma} * dmu * dsigma
        ddsig = -dm**2 / (2*sig) + ds**2 / sig  # Gamma terms
        return np.array([dm, ds, ddmu, ddsig])

    k1 = derivs(state)
    k2 = derivs(state + 0.5*dt*k1)
    k3 = derivs(state + 0.5*dt*k2)
    k4 = derivs(state + dt*k3)
    return state + (dt/6) * (k1 + 2*k2 + 2*k3 + k4)

Natural Gradient Implementation

Side-by-side comparison of Euclidean and natural gradient descent minimizing $D_{\mathrm{KL}}(\mathcal{N}(0,1) \,\|\, \mathcal{N}(\mu, \sigma^2))$:

def gradient_descent(mu, sigma, target_mu, target_sigma, lr, natural=False, steps=100):
    """Euclidean or natural gradient descent on KL divergence."""
    trajectory = [(mu, sigma)]
    for _ in range(steps):
        # Euclidean gradient of D_KL(target || model)
        grad_mu = (mu - target_mu) / sigma**2
        grad_sigma = 1/sigma - (target_sigma**2 + (target_mu - mu)**2) / sigma**3

        if natural:
            # Fisher metric inverse: g^{-1} = diag(sigma^2, sigma^2/2)
            nat_mu = sigma**2 * grad_mu
            nat_sigma = (sigma**2 / 2) * grad_sigma
            mu -= lr * nat_mu
            sigma -= lr * nat_sigma
        else:
            mu -= lr * grad_mu
            sigma -= lr * grad_sigma

        sigma = max(sigma, 0.01)  # Keep sigma positive
        trajectory.append((mu, sigma))
    return trajectory

Fisher-Rao Distance Matrix

Pairwise Fisher-Rao distances between Gaussians using Rao’s formula:

def fisher_rao_distance(mu1, s1, mu2, s2):
    """Closed-form Fisher-Rao distance for univariate Gaussians."""
    return np.sqrt(2) * np.arccosh(
        1 + ((mu1 - mu2)**2 + 2*(s1 - s2)**2) / (4 * s1 * s2)
    )

Note that this formula uses the half-plane metric $ds^2 = \frac{1}{\sigma^2}(d\mu^2 + 2\,d\sigma^2)$, which includes the factor of $2$ in the $d\sigma^2$ term. The $\text{arccosh}$ reflects the hyperbolic nature of the geometry.

---

Information Geometry in Machine Learning

Information geometry provides the natural mathematical framework for understanding and improving several core machine learning algorithms.

Natural Gradient Descent

Standard gradient descent updates parameters as $\theta_{t+1} = \theta_t - \eta \, \nabla L(\theta_t)$, using the Euclidean gradient. But the Euclidean gradient depends on the parameterization: reparameterizing the same model changes the gradient direction. This is undesirable — the “steepest descent” direction should be an intrinsic property of the model, not an artifact of how we chose to write down its parameters.

Amari’s natural gradient (1998) fixes this by using the Fisher-Rao metric:

$\theta_{t+1} = \theta_t - \eta \, g^{-1}(\theta_t) \, \nabla L(\theta_t)$

The natural gradient $\tilde{\nabla} L = g^{-1} \nabla L$ is the steepest descent direction in the Riemannian metric defined by the Fisher information. It is:

• Reparameterization invariant: changing coordinates $\theta \to \phi(\theta)$ does not change the natural gradient direction (it transforms covariantly).
• Asymptotically efficient: for maximum likelihood estimation, natural gradient descent converges to the optimal rate.
• Follows geodesics approximately: the natural gradient flow traces out curves that are close to Fisher-Rao geodesics.

Variational Inference as $m$-Projection

Variational inference minimizes $D_{\mathrm{KL}}(q \,\|\, p^*)$ over a variational family $\mathcal{Q}$ to approximate an intractable posterior $p^*$. In information-geometric terms, this is the $m$-projection of $p^*$ onto $\mathcal{Q}$: the distribution in $\mathcal{Q}$ closest to $p^*$ in the $m$-connection sense.

When $\mathcal{Q}$ is an exponential family (a dually flat manifold), the generalized Pythagorean theorem guarantees:

$D_{\mathrm{KL}}(q \,\|\, p^*) = D_{\mathrm{KL}}(q \,\|\, q^*) + D_{\mathrm{KL}}(q^* \,\|\, p^*)$

where $q^*$ is the optimal variational approximation. The second term is the irreducible approximation error (determined by the expressiveness of $\mathcal{Q}$), and the first term is what the optimization eliminates.

Adam as Approximate Natural Gradient

The Adam optimizer (Kingma & Ba, 2015) maintains running estimates of the first and second moments of the gradient. The second moment estimate $v_t \approx \mathbb{E}[(\nabla L)^2]$ is a diagonal approximation to the Fisher information matrix: $\text{diag}(v_t) \approx \text{diag}(g(\theta))$. The Adam update

$\theta_{t+1} = \theta_t - \eta \, \frac{m_t}{\sqrt{v_t} + \epsilon}$

is therefore an approximate natural gradient step with a diagonal Fisher matrix.

K-FAC (Martens & Grosse, 2015) improves on this by using a block-diagonal, Kronecker-factored approximation to the full Fisher matrix. For a neural network layer with input $a$ and output gradient $g$, K-FAC approximates the Fisher block as $\mathbb{E}[a a^T] \otimes \mathbb{E}[g g^T]$, which is far cheaper to invert than the full Fisher matrix while capturing more curvature structure than Adam’s diagonal.

Optimal Transport Connections

The Fisher-Rao metric and the Wasserstein distance define different geometries on the space of distributions:

• Fisher-Rao measures informational distance: how distinguishable are two distributions from finite samples?
• Wasserstein measures physical distance: what is the cost of transporting mass from one distribution to the other?

Otto (2001) showed that the Wasserstein space carries a formal Riemannian structure where the gradient flow of the KL divergence is the Fokker-Planck equation. The interplay between these two geometries — informational and physical — is an active research frontier connecting information geometry to optimal transport theory.

Loss Landscape Curvature

The curvature of the loss landscape at a minimum affects generalization. The “flat minima” conjecture (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017) suggests that minima occupying large, flat regions of the loss surface generalize better than sharp minima. The Fisher information matrix at convergence is related to the Hessian of the loss, and its eigenspectrum characterizes the sharpness of the minimum.

Specifically, for a model trained with maximum likelihood, the Fisher information matrix equals the expected Hessian of the negative log-likelihood. The eigenvalues of $g(\theta^*)$ at the converged parameters $\theta^*$ measure the curvature in each direction: large eigenvalues correspond to sharp directions, and the Spectral Theorem guarantees that these principal curvature directions exist and are orthogonal.

---

Connections & Further Reading

Information Geometry & Fisher Metric is the capstone of the Differential Geometry track. It connects back to every topic in the track and forward to applications across the curriculum:

Connected Topic	Domain	Relationship
Smooth Manifolds	Differential Geometry	Parameter spaces as smooth manifolds; tangent spaces spanned by score functions
Riemannian Geometry	Differential Geometry	Fisher metric as Riemannian metric; Levi-Civita as $\alpha=0$ connection
Geodesics & Curvature	Differential Geometry	Fisher-Rao geodesics; curvature $K=-1/2$ for Gaussians; Gauss-Bonnet theorem
The Spectral Theorem	Linear Algebra	Eigendecomposition of Fisher matrix reveals principal information directions
PCA & Low-Rank Approximation	Linear Algebra	Natural gradient as Fisher preconditioning; covariance structure
Persistent Homology	Topology & TDA	Euler characteristic in Gauss-Bonnet for statistical manifolds
Shannon Entropy & Mutual Information	Information Theory	The entropy and mutual information quantities are developed rigorously on the Information Theory track. KL divergence $D_{KL}(p \| q) = H(p,q) - H(p)$ is the divergence whose Hessian gives the Fisher metric. The KL divergence and its f-divergence generalizations are developed in KL Divergence & f-Divergences.

Completing the Differential Geometry Track

With this topic, the four-part Differential Geometry track is complete:

1. Smooth Manifolds (intermediate) — the foundational structure: charts, tangent spaces, and smooth maps
2. Riemannian Geometry (advanced) — metric tensors, connections, and parallel transport
3. Geodesics & Curvature (intermediate) — geodesic equations, curvature tensors, and the Gauss-Bonnet theorem
4. Information Geometry & Fisher Metric (advanced) — the Fisher metric on statistical manifolds, $\alpha$-connections, and natural gradient descent

The track moves from abstract manifold structure through Riemannian geometry to the concrete, application-rich setting of statistical manifolds — where the geometric machinery built in the first three topics has direct consequences for machine learning.