Created: October 21, 2022
Modified: October 22, 2022

replica trick

This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.

If a model with data $x$ has normalizing constant $Z(x)$ , then the replica trick says that

\mathbb{E}[\log Z] = \lim_{n\to 0} \frac{1}{n}\log \mathbb{E}[Z^n]

This allows us to analyze the average log-normalizer (generally hard to compute directly) in terms of the limiting behavior of $n$ independent 'replicas' of the system.

It is a bit mathematically squishy because one typically works out an expression for $\mathbb{E}[Z^n]$ using the product of $n$ independent replicas, assuming that $n$ is an integer. Taking the limit of this expression as $n$ goes to zero relies on analytic continuation, which often works but is not always formally justified.

Derivation

Begin by considering the first-order Taylor expansion:

\begin{align*}x^n &= \exp(n \log x)\\&=\exp(0) + n\log x \cdot \exp'(0) + \frac{(n \log x)^2}{2}\exp''(0) + \cdots\\&=1 + n\log x + \frac{(n \log x)^2}{2} + \cdots\\&\approx 1 + n \log x\text{ for small $n$}.\end{align*}

This implies the identity

\log x = \lim_{n\to 0}\frac{x^n - 1}{n}

and thus

\mathbb{E}[\log Z] = \lim_{n\to 0} \frac{\mathbb{E}[Z^n] - 1}{n}.

To proceed, we will use another first-order Taylor expansion,

\begin{align*}\log(1 + x) &= \log(1) + x \cdot \log'(1) + \frac{x^2}{2}\log''(1) + \cdots\\&= 0 + x + \frac{x^2}{2} + \cdots\\ &\approx x \text{ for small $x$}\end{align*}

which, taking $x = \mathbb{E}[Z^n] - 1$ (and noting that this approaches zero for small $n$ ), in particular implies

\mathbb{E}[Z^n] - 1 \approx \log(\mathbb{E}[Z^n]).

Applying this substitution to the result above recovers the final expression,

\mathbb{E}[\log Z] = \lim_{n\to 0} \frac{1}{n}\log \mathbb{E}[Z^n].

replica trick

Derivation

References:

Meta