Excyst/Formulas
1,038 formulas, from probability distributions and statistics to linear algebra, optimisation, deep learning, signal processing and finance. Each has LaTeX to paste into your paper, its symbols defined, and a worked example that is checked in code.
1,038 formulas
bounded-difference martingales concentrate
where |Mᵢ − Mᵢ₋₁| ≤ cᵢ
Worked example: t = 10, c = [100 values] → 1.21306 ✓ checked
See also: McDiarmid's inequality · Hoeffding's inequality
P\big(|M_n-M_0|\ge t\big)\le2\exp\!\Big(-\frac{t^2}{2\sum_ic_i^2}\Big)P(A|B) = P(B|A)·P(A) / P(B)
where P(A) prior, P(B|A) likelihood, P(A|B) posterior, P(B) evidence
With the law of total probability, P(B) = Σ P(B|Aᵢ)P(Aᵢ).
Worked example: pba = 0.9, pa = 0.01, pbna = 0.05 → 0.153846 ✓ checked
See also: law of total probability · maximum a posteriori (Gaussian mean, Gaussian prior)
P(A\mid B) = \frac{P(B\mid A)\,P(A)}{P(B)}P(X = x) = pˣ (1−p)¹⁻ˣ
where p = probability of success
Mean p, variance p(1−p).
Worked example: x = 1, p = 0.3 → 0.3 ✓ checked
See also: binomial distribution
P(X=x) = p^{x}(1-p)^{1-x},\quad x\in\{0,1\}E[X] = p
Verified against summation of the mass function.
Worked example: p = 0.3, x0 = 1 → 0.3 ✓ checked
See also: Bernoulli distribution: variance
\mathbb E[X]=pVar(X) = p(1 − p)
Verified against summation of the mass function.
Worked example: p = 0.3, x0 = 1 → 0.21 ✓ checked
See also: Bernoulli distribution: mean
\operatorname{Var}(X)=p(1-p)f(x) = x^(α−1) (1−x)^(β−1) / B(α,β)
where α, β > 0; x ∈ [0,1]
Conjugate prior for the Bernoulli/binomial success probability.
Worked example: a = 2, b = 3, x = 0.5 → 1.5 ✓ checked
See also: Bernoulli distribution · binomial distribution · Bayes' theorem
f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)},\quad B(\alpha,\beta)=\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}E[X] = α/(α+β)
Verified against numerical integration of the density.
Worked example: al = 2, be = 5, x0 = 0.3 → 0.285714 ✓ checked
See also: beta distribution: variance
\mathbb E[X]=\frac{\alpha}{\alpha+\beta}Var(X) = αβ / ((α+β)²(α+β+1))
Verified against numerical integration of the density.
Worked example: al = 2, be = 5, x0 = 0.3 → 0.0255102 ✓ checked
See also: beta distribution: mean
\operatorname{Var}(X)=\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}E[X] = nα/(α+β)
Verified against summation of the mass function.
Worked example: n = 10, al = 2, be = 3, x0 = 4 → 4 ✓ checked
See also: beta-binomial distribution: variance
\mathbb E[X]=\frac{n\alpha}{\alpha+\beta}P(X = k) = C(n,k) B(k+α, n−k+β) / B(α, β)
A binomial whose success probability is itself Beta(α, β): overdispersed relative to the binomial.
Worked example: n = 10, al = 2, be = 3, x0 = 4 → 0.13986 ✓ checked
See also: beta-binomial distribution: mean
P(X=k)=\binom nk\frac{B(k+\alpha,\,n-k+\beta)}{B(\alpha,\beta)}Var(X) = nαβ(α+β+n) / ((α+β)²(α+β+1))
Verified against summation of the mass function.
Worked example: n = 10, al = 2, be = 3, x0 = 4 → 6 ✓ checked
See also: beta-binomial distribution: mean
\operatorname{Var}(X)=\frac{n\alpha\beta(\alpha+\beta+n)}{(\alpha+\beta)^2(\alpha+\beta+1)}P(X = k) = C(n,k) pᵏ (1−p)ⁿ⁻ᵏ
where n trials, k successes, p = success probability
Mean np, variance np(1−p).
Worked example: n = 10, k = 3, p = 0.5 → 0.117188 ✓ checked
See also: Bernoulli distribution · Poisson distribution
P(X=k) = \binom{n}{k}p^{k}(1-p)^{n-k}E[X] = np
Verified against summation of the mass function.
Worked example: n = 10, p = 0.3, x0 = 3 → 3 ✓ checked
See also: binomial distribution: variance
\mathbb E[X]=npM(t) = (1 − p + p eᵗ)ⁿ
Worked example: n = 10, p = 0.3, t = 0.4 → 3.9601 ✓ checked
See also: binomial distribution: mean
M_X(t)=\big(1-p+pe^{t}\big)^nVar(X) = np(1 − p)
Verified against summation of the mass function.
Worked example: n = 10, p = 0.3, x0 = 3 → 2.1 ✓ checked
See also: binomial distribution: mean
\operatorname{Var}(X)=np(1-p)πₖ = π₀ Π λ_{i−1}/μᵢWorked example: rho = 0.8, k = 3 → 0.1024 ✓ checked
See also: M/M/1 queue
\pi_k=\pi_0\prod_{i=1}^{k}\frac{\lambda_{i-1}}{\mu_i}\quad\Rightarrow\quad M/M/1:\ \pi_k=(1-\rho)\rho^kincrements are independent N(0, t − s); Cov(Wₛ, Wₜ) = min(s, t)
Continuous but nowhere differentiable; the limit of a scaled random walk.
Worked example: s = 2, t = 5 → [3, 2] ✓ checked
See also: random walk · geometric Brownian motion
W_0=0,\quad W_t-W_s\sim\mathcal N(0,\,t-s),\quad \operatorname{Cov}(W_s,W_t)=\min(s,t)P(X = k) = pₖ
One draw from K classes; the output of a softmax classifier. Log-likelihood of one observation is log pₖ.
Worked example: p = [0.2, 0.5, 0.3], k = 1 → 0.5 ✓ checked
See also: softmax · multinomial distribution
P(X=k)=p_k,\quad \sum_k p_k=1F(x) = ½ + arctan((x − x₀)/γ)/π
Verified against numerical integration of the density up to x.
Worked example: loc = 0.5, g = 2, x0 = 1 → 0.577789 ✓ checked
See also: Cauchy distribution: density
F(x)=\tfrac12+\frac1\pi\arctan\!\Big(\frac{x-x_0}{\gamma}\Big)f(x) = 1 / (πγ [1 + ((x − x₀)/γ)²])
No mean and no variance (the integrals diverge): sample averages of Cauchy data do not settle down.
Worked example: loc = 0.5, g = 2, x0 = 1 → 0.149793 ✓ checked
See also: Cauchy distribution: cumulative distribution function
f(x)=\frac{1}{\pi\gamma\Big[1+\big(\frac{x-x_0}{\gamma}\big)^2\Big]}P(A₁,…,Aₙ) = Πᵢ P(Aᵢ | A₁,…,Aᵢ₋₁)
The basis of autoregressive language models.
Worked example: p = [0.5, 0.4, 0.2] → 0.04 ✓ checked
See also: Bayes' theorem
P(A_1,\dots,A_n) = \prod_{i=1}^{n} P(A_i \mid A_1,\dots,A_{i-1})f_Y(y) = f_X(g⁻¹(y))·|d g⁻¹/dy|
Example: Y = 2X for X ~ Exp(1) gives f_Y = ½ e^(−y/2).
Worked example: y = 2 → 0.18394 ✓ checked
See also: normalising flow change of variables
Y=g(X)\ \Rightarrow\ f_Y(y)=f_X\big(g^{-1}(y)\big)\Big|\frac{d\,g^{-1}(y)}{dy}\Big|the n-step transition matrix is the n-th power of P
Worked example: P = [[0.9, 0.1], [0.5, 0.5]] → [[0.86, 0.14], [0.7, 0.3]] ✓ checked
See also: stationary distribution of a Markov chain
P^{(m+n)}=P^{(m)}P^{(n)},\qquad P^{(n)}=P^nP(|X − μ| ≥ kσ) ≤ 1/k²
where k > 0
Needs only a finite variance.
Worked example: k = 2 → 0.25 ✓ checked
See also: Markov's inequality
P\big(|X-\mu|\ge k\sigma\big)\le \frac{1}{k^2}F(x) = 1 − e^(−x/2)(1 + x/2) for k = 4
Verified against numerical integration of the density up to x.
Worked example: k = 4, x0 = 3 → 0.442175 ✓ checked
See also: chi-square distribution: density
F(x)=1-e^{-x/2}\Big(1+\frac x2\Big)\quad(k=4)f(x) = x^(k/2−1) e^(−x/2) / (2^(k/2) Γ(k/2))
The sum of k squared independent standard normals. Gamma with shape k/2 and scale 2.
Worked example: k = 4, x0 = 3 → 0.167348 ✓ checked
See also: chi-square distribution: mean
f(x)=\frac{x^{k/2-1}e^{-x/2}}{2^{k/2}\Gamma(k/2)}E[X] = k
Verified against numerical integration of the density.
Worked example: k = 4, x0 = 3 → 4 ✓ checked
See also: chi-square distribution: variance
\mathbb E[X]=kM(t) = (1 − 2t)^(−k/2)
Worked example: k = 4, t = 0.2 → 2.77778 ✓ checked
See also: chi-square distribution: mean
M_X(t)=(1-2t)^{-k/2}\quad(t<1/2)Var(X) = 2k
Verified against numerical integration of the density.
Worked example: k = 4, x0 = 3 → 8 ✓ checked
See also: chi-square distribution: mean
\operatorname{Var}(X)=2kE S = λt·E[Y]; Var S = λt·E[Y²]
Insurance claims and call-centre loads.
Worked example: lam = 3, t = 2, EY = 10, EY2 = 150 → [60, 900] ✓ checked
See also: Wald's identity
S_t=\sum_{i=1}^{N_t}Y_i:\quad \mathbb E S_t=\lambda t\,\mathbb E Y,\quad \operatorname{Var}S_t=\lambda t\,\mathbb E[Y^2]F(x) = (x − a)/(b − a)
Verified against numerical integration of the density up to x.
Worked example: a = 1, b = 4, x0 = 2 → 0.333333 ✓ checked
See also: continuous uniform distribution: density
F(x)=\frac{x-a}{b-a}f(x) = 1/(b − a) on [a, b]
Every value in [a, b] is equally likely. Maximum entropy on a bounded interval.
Worked example: a = 1, b = 4, x0 = 2 → 0.333333 ✓ checked
See also: continuous uniform distribution: mean
f(x)=\frac{1}{b-a}\quad(a\le x\le b)μ = (a + b)/2
Verified against numerical integration of the density.
Worked example: a = 1, b = 4, x0 = 2 → 2.5 ✓ checked
See also: continuous uniform distribution: variance
\mu=\frac{a+b}{2}σ² = (b − a)²/12
Verified against numerical integration of the density.
Worked example: a = 1, b = 4, x0 = 2 → 0.75 ✓ checked
See also: continuous uniform distribution: mean
\sigma^2=\frac{(b-a)^2}{12}Cov(X,Y) = E[XY] − E[X]E[Y]
Zero covariance does not imply independence. Cov(X,X) = Var(X).
Worked example: x = [1, 2, 3, 4], y = [2, 4, 5, 9] → 2.75 ✓ checked
See also: Pearson correlation · variance and standard deviation
\operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\,\mathbb{E}[Y]K(t) = ln M(t); κ₁ = mean, κ₂ = variance
Cumulants of independent sums add. For a normal, all cumulants above the second vanish.
Worked example: mu = 1, s = 2 → [1, 4] ✓ checked
See also: moments from the moment generating function
K_X(t)=\ln M_X(t),\quad \kappa_1=\mathbb E[X],\ \kappa_2=\operatorname{Var}(X)f_{X+Y} = f_X ∗ f_YTwo Uniform(0, 1) give the triangular density.
Worked example: z = 1 → 1 ✓ checked
See also: discrete convolution · sum of independent normals
f_{X+Y}(z)=\int f_X(x)\,f_Y(z-x)\,dxf₍ₖ₎ = n!/((k−1)!(n−k)!) F^(k−1) (1 − F)^(n−k) f
Worked example: n = 5, k = 3, x = 0.5 → 1.875 ✓ checked
See also: order statistics of uniforms
f_{(k)}(x)=\frac{n!}{(k-1)!\,(n-k)!}F(x)^{k-1}\big(1-F(x)\big)^{n-k}f(x)πᵢPᵢⱼ = πⱼPⱼᵢ
The condition that Metropolis–Hastings enforces so that π is the stationary distribution.
Worked example: P = [[0.9, 0.1], [0.5, 0.5]], pi = [0.833333, 0.166667] → 1.38778e-17 ✓ checked
See also: Metropolis–Hastings acceptance · stationary distribution of a Markov chain
\pi_iP_{ij}=\pi_jP_{ji}\ \ \Rightarrow\ \ \boldsymbol\pi P=\boldsymbol\pif(p) = Γ(α₀)/ΠΓ(αₖ) · Π pₖ^(αₖ−1)
Mean αₖ/α₀. The conjugate prior of the categorical and multinomial; the multivariate beta.
Worked example: al = [2, 3, 4], p = [0.2, 0.3, 0.5] → 7.56 ✓ checked
See also: beta distribution: mean · multinomial distribution
f(\mathbf p)=\frac{\Gamma(\alpha_0)}{\prod_k\Gamma(\alpha_k)}\prod_k p_k^{\alpha_k-1},\quad \alpha_0=\sum_k\alpha_kE[X] = (n + 1)/2
Verified against summation of the mass function.
Worked example: n = 6, x0 = 3 → 3.5 ✓ checked
See also: discrete uniform distribution: variance
\mathbb E[X]=\frac{n+1}{2}P(X = k) = 1/n
A fair die has n = 6.
Worked example: n = 6, x0 = 3 → 0.166667 ✓ checked
See also: discrete uniform distribution: mean
P(X=k)=\frac1n,\ k=1,\dots,nVar(X) = (n² − 1)/12
Verified against summation of the mass function.
Worked example: n = 6, x0 = 3 → 2.91667 ✓ checked
See also: discrete uniform distribution: mean
\operatorname{Var}(X)=\frac{n^2-1}{12}E[X] = Σₓ x·p(x) (continuous: ∫ x f(x) dx)
where p(x) = probability mass, f(x) = density
Linear: E[aX + bY] = aE[X] + bE[Y], always, even when X and Y are dependent.
Worked example: x = [1, 2, 3], p = [0.2, 0.5, 0.3] → 2.1 ✓ checked
See also: variance and standard deviation · law of iterated expectation
\mathbb{E}[X] = \sum_x x\,p(x) \qquad \mathbb{E}[X] = \int x\,f(x)\,dxf(x) = λ e^(−λx) F(x) = 1 − e^(−λx)
where λ = rate; mean 1/λ
Memoryless: P(X > s+t | X > s) = P(X > t).
Worked example: lam = 2, x = 0.5 → 0.735759 ✓ checked
See also: Poisson distribution
f(x) = \lambda e^{-\lambda x},\ x\ge 0 \qquad F(x) = 1-e^{-\lambda x}E[X] = 1/λ
Verified against numerical integration of the density.
Worked example: lam = 2, x0 = 0.5 → 0.5 ✓ checked
See also: exponential distribution: variance
\mathbb E[X]=\frac1\lambdaM(t) = λ/(λ − t)
Worked example: lam = 2, t = 0.5 → 1.33333 ✓ checked
See also: exponential distribution: mean
M_X(t)=\frac{\lambda}{\lambda-t}\quad(t<\lambda)Var(X) = 1/λ²
Verified against numerical integration of the density.
Worked example: lam = 2, x0 = 0.5 → 0.25 ✓ checked
See also: exponential distribution: mean
\operatorname{Var}(X)=\frac1{\lambda^2}the maximum of n exponentials is asymptotically Gumbel
Worked example: n = 100000, x = 0.5 → 0.545238 ✓ checked
See also: Gumbel (extreme value type I) distribution: density
P\big(\max_{i\le n}X_i-\ln n\le x\big)\to\exp\!\big(-e^{-x}\big)\qquad(X_i\sim\mathrm{Exp}(1))f(x) = √( (d₁x)^d₁ d₂^d₂ / (d₁x+d₂)^(d₁+d₂) ) / (x B(d₁/2, d₂/2))
The ratio of two scaled chi-squares; the null distribution of the ANOVA F statistic.
Worked example: d1 = 5, d2 = 10, x0 = 1.5 → 0.286459 ✓ checked
See also: F distribution distribution: mean
f(x)=\frac{1}{x\,B(d_1/2,d_2/2)}\sqrt{\frac{(d_1x)^{d_1}\,d_2^{d_2}}{(d_1x+d_2)^{d_1+d_2}}}E[X] = d₂/(d₂ − 2)
Verified against numerical integration of the density.
Worked example: d1 = 5, d2 = 10, x0 = 1.5 → 1.25 ✓ checked
See also: F distribution distribution: variance
\mathbb E[X]=\frac{d_2}{d_2-2}\ \ (d_2>2)Var(X) = 2d₂²(d₁+d₂−2) / (d₁(d₂−2)²(d₂−4))
Verified against numerical integration of the density.
Worked example: d1 = 5, d2 = 10, x0 = 1.5 → 1.35416 ✓ checked
See also: F distribution distribution: mean
\operatorname{Var}(X)=\frac{2d_2^2(d_1+d_2-2)}{d_1(d_2-2)^2(d_2-4)}\ \ (d_2>4)E[T] = i(N − i)
Worked example: i = 10, N = 30 → 200 ✓ checked
See also: gambler's ruin
\mathbb E[T]=i(N-i)\quad\text{for a symmetric walk on }\{0,\dots,N\}\text{ started at }iF(x) = 1 − e^(−x/θ) Σ_{j<k} (x/θ)ʲ/j! (integer k, the Erlang case)Verified against numerical integration of the density up to x.
Worked example: k = 3, th = 2, x0 = 4 → 0.323324 ✓ checked
See also: gamma distribution: density
F(x)=1-e^{-x/\theta}\sum_{j=0}^{k-1}\frac{(x/\theta)^j}{j!}\ \ (k\in\mathbb N)f(x) = x^(k−1) e^(−x/θ) / (Γ(k) θᵏ)
Shape k, scale θ. The sum of k independent exponentials (mean θ) is gamma. Some texts use rate β = 1/θ.
Worked example: k = 3, th = 2, x0 = 4 → 0.135335 ✓ checked
See also: gamma distribution: mean
f(x)=\frac{x^{k-1}e^{-x/\theta}}{\Gamma(k)\,\theta^{k}}E[X] = kθ
Verified against numerical integration of the density.
Worked example: k = 3, th = 2, x0 = 4 → 6 ✓ checked
See also: gamma distribution: variance
\mathbb E[X]=k\thetaM(t) = (1 − θt)^(−k)
The product of independent gamma MGFs with common θ adds the shapes.
Worked example: k = 3, th = 2, t = 0.2 → 4.62963 ✓ checked
See also: gamma distribution: mean
M_X(t)=(1-\theta t)^{-k}\quad(t<1/\theta)Var(X) = kθ²
Verified against numerical integration of the density.
Worked example: k = 3, th = 2, x0 = 4 → 12 ✓ checked
See also: gamma distribution: mean
\operatorname{Var}(X)=k\theta^2S_t = S₀ exp((μ − σ²/2)t + σW_t); E S_t = S₀ e^(μt)
The Black–Scholes model of a stock price.
Worked example: S0 = 100, mu = 0.05, s = 0.2, t = 1 → [105.127, 451.029] ✓ checked
See also: Itô's lemma · Black–Scholes option price
dS=\mu S\,dt+\sigma S\,dW,\quad S_t=S_0\exp\!\big((\mu-\tfrac12\sigma^2)t+\sigma W_t\big),\quad \mathbb E S_t=S_0e^{\mu t}F(k) = 1 − (1−p)ᵏ
Verified against summation of the mass function up to k.
Worked example: p = 0.3, x0 = 4 → 0.7599 ✓ checked
See also: geometric distribution: probability mass
F(k)=1-(1-p)^kE[X] = 1/p
Verified against summation of the mass function.
Worked example: p = 0.3, x0 = 4 → 3.33333 ✓ checked
See also: geometric distribution: variance
\mathbb E[X]=\frac1pP(X = k) = (1−p)^(k−1) p
The number of trials up to and including the first success. Memoryless.
Worked example: p = 0.3, x0 = 4 → 0.1029 ✓ checked
See also: geometric distribution: mean
P(X=k)=(1-p)^{k-1}p,\ k\ge1Var(X) = (1 − p)/p²
Verified against summation of the mass function.
Worked example: p = 0.3, x0 = 4 → 7.77778 ✓ checked
See also: geometric distribution: mean
\operatorname{Var}(X)=\frac{1-p}{p^2}F(x) = exp(−e^(−(x−μ)/β))
Verified against numerical integration of the density up to x.
Worked example: mu = 0.5, be = 1.5, x0 = 1 → 0.488444 ✓ checked
See also: Gumbel (extreme value type I) distribution: density
F(x)=\exp\!\big(-e^{-(x-\mu)/\beta}\big)f(x) = (1/β) exp(−(z + e^(−z))), z = (x−μ)/β
Distribution of the maximum of many samples; the Gumbel-softmax trick samples from it.
Worked example: mu = 0.5, be = 1.5, x0 = 1 → 0.233323 ✓ checked
See also: Gumbel (extreme value type I) distribution: mean
f(x)=\frac1\beta\exp\!\big(-(z+e^{-z})\big),\ z=\frac{x-\mu}{\beta}E[X] = μ + βγ (γ ≈ 0.5772, Euler–Mascheroni)
Verified against numerical integration of the density.
Worked example: mu = 0.5, be = 1.5, x0 = 1 → 1.36582 ✓ checked
See also: Gumbel (extreme value type I) distribution: variance
\mathbb E[X]=\mu+\beta\gamma_EVar(X) = π²β²/6
Verified against numerical integration of the density.
Worked example: mu = 0.5, be = 1.5, x0 = 1 → 3.7011 ✓ checked
See also: Gumbel (extreme value type I) distribution: mean
\operatorname{Var}(X)=\frac{\pi^2\beta^2}{6}f(x) = √2/(σ√π) · e^(−x²/2σ²)
|Z| for Z ~ N(0, σ²).
Worked example: s = 2, x0 = 1 → 0.352065 ✓ checked
See also: half-normal distribution: mean
f(x)=\frac{\sqrt2}{\sigma\sqrt\pi}e^{-x^2/2\sigma^2},\ x\ge0E[X] = σ√(2/π)
Verified against numerical integration of the density.
Worked example: s = 2, x0 = 1 → 1.59577 ✓ checked
See also: half-normal distribution: variance
\mathbb E[X]=\sigma\sqrt{\tfrac2\pi}Var(X) = σ²(1 − 2/π)
Verified against numerical integration of the density.
Worked example: s = 2, x0 = 1 → 1.45352 ✓ checked
See also: half-normal distribution: mean
\operatorname{Var}(X)=\sigma^2\Big(1-\frac2\pi\Big)P( X̄ − E[X̄] ≥ t ) ≤ exp( −2n t² / (b−a)² )
where Xᵢ independent, in [a, b]; n samples
Gives finite-sample confidence bounds for bounded variables.
Worked example: n = 100, t = 0.1, r = 1 → 0.135335 ✓ checked
See also: Chebyshev's inequality
P\big(\bar X-\mathbb{E}\bar X\ge t\big)\le \exp\!\left(-\frac{2nt^2}{(b-a)^2}\right)E[X] = nK/N
Verified against summation of the mass function.
Worked example: N = 50, K = 10, n = 8, x0 = 2 → 1.6 ✓ checked
See also: hypergeometric distribution: variance
\mathbb E[X]=\frac{nK}{N}P(X = k) = C(K,k) C(N−K, n−k) / C(N, n)
Successes when drawing n items without replacement from N containing K successes. The factor (N−n)/(N−1) is the finite-population correction.
Worked example: N = 50, K = 10, n = 8, x0 = 2 → 0.321725 ✓ checked
See also: hypergeometric distribution: mean
P(X=k)=\frac{\binom Kk\binom{N-K}{n-k}}{\binom Nn}Var(X) = n (K/N)(1 − K/N)(N − n)/(N − 1)
Verified against summation of the mass function.
Worked example: N = 50, K = 10, n = 8, x0 = 2 → 1.09714 ✓ checked
See also: hypergeometric distribution: mean
\operatorname{Var}(X)=n\frac KN\Big(1-\frac KN\Big)\frac{N-n}{N-1}f(x) = βᵅ/Γ(α) · x^(−α−1) e^(−β/x)
Conjugate prior for the variance of a normal.
Worked example: al = 4, be = 3, x0 = 1 → 0.672125 ✓ checked
See also: inverse gamma distribution: mean
f(x)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}x^{-\alpha-1}e^{-\beta/x}E[X] = β/(α − 1)
Verified against numerical integration of the density.
Worked example: al = 4, be = 3, x0 = 1 → 1 ✓ checked
See also: inverse gamma distribution: variance
\mathbb E[X]=\frac{\beta}{\alpha-1}\ \ (\alpha>1)Var(X) = β² / ((α−1)²(α−2))
Verified against numerical integration of the density.
Worked example: al = 4, be = 3, x0 = 1 → 0.5 ✓ checked
See also: inverse gamma distribution: mean
\operatorname{Var}(X)=\frac{\beta^2}{(\alpha-1)^2(\alpha-2)}\ \ (\alpha>2)df = (f_t + μf_x + ½σ²f_xx)dt + σf_x dW
The extra ½σ²f_xx term is why ln S has drift μ − σ²/2 for geometric Brownian motion.
Worked example: mu = 0.05, s = 0.2 → 0.03 ✓ checked
See also: geometric Brownian motion
df(t,X)=\Big(f_t+\mu f_x+\tfrac12\sigma^2f_{xx}\Big)dt+\sigma f_x\,dWf(E[X]) ≤ E[f(X)] for convex f
Reverses for concave f. Gives KL ≥ 0.
Worked example: x = [1, 3], p = [0.5, 0.5] → 1 ✓ checked
See also: Kullback–Leibler divergence
f(\mathbb{E}[X])\le \mathbb{E}[f(X)]\quad (f\ \text{convex})marginalise, divide, and independence factorises
Worked example: pxy = 0.12, py = 0.4 → 0.3 ✓ checked
See also: Bayes' theorem
f_X(x)=\int f_{X,Y}(x,y)\,dy,\qquad f_{X\mid Y}(x\mid y)=\frac{f_{X,Y}(x,y)}{f_Y(y)},\qquad X\perp Y\iff f_{X,Y}=f_Xf_YF(x) = 1 − ½ e^(−(x−μ)/b) for x ≥ μ
Verified against numerical integration of the density up to x.
Worked example: mu = 1, b = 2, x0 = 2 → 0.696735 ✓ checked
See also: Laplace (double exponential) distribution: density
F(x)=1-\tfrac12e^{-(x-\mu)/b}\quad(x\ge\mu)f(x) = e^(−|x−μ|/b) / (2b)
The Bayesian reading of the L1 penalty (lasso) is a Laplace prior on the weights.
Worked example: mu = 1, b = 2, x0 = 2 → 0.151633 ✓ checked
See also: Laplace (double exponential) distribution: mean
f(x)=\frac{1}{2b}e^{-|x-\mu|/b}E[X] = μ
Verified against numerical integration of the density.
Worked example: mu = 1, b = 2, x0 = 2 → 1 ✓ checked
See also: Laplace (double exponential) distribution: variance
\mathbb E[X]=\muVar(X) = 2b²
Verified against numerical integration of the density.
Worked example: mu = 1, b = 2, x0 = 2 → 8 ✓ checked
See also: Laplace (double exponential) distribution: mean
\operatorname{Var}(X)=2b^2E[Y] = E[ E[Y|X] ]
Also called the tower property.
Worked example: m = [0, 2], p = [0.5, 0.5] → 1 ✓ checked
See also: expected value · law of total variance
\mathbb{E}[Y] = \mathbb{E}\big[\mathbb{E}[Y\mid X]\big]P(B) = Σᵢ P(B|Aᵢ)·P(Aᵢ)
where A₁…Aₙ partition the sample space
Worked example: pba = [0.9, 0.05], pa = [0.01, 0.99] → 0.0585 ✓ checked
See also: Bayes' theorem
P(B) = \sum_i P(B\mid A_i)\,P(A_i)Var(Y) = E[Var(Y|X)] + Var(E[Y|X])
Within-group variance plus between-group variance.
Worked example: v = [1, 4], m = [0, 2], p = [0.5, 0.5] → 3.5 ✓ checked
See also: variance and standard deviation
\operatorname{Var}(Y) = \mathbb{E}[\operatorname{Var}(Y\mid X)] + \operatorname{Var}(\mathbb{E}[Y\mid X])F(x) = ½[1 + erf((ln x − μ)/(σ√2))]
Verified against numerical integration of the density up to x.
Worked example: mu = 0, s = 0.5, x0 = 1.5 → 0.791297 ✓ checked
See also: log-normal distribution: density
F(x)=\tfrac12\Big[1+\operatorname{erf}\!\Big(\frac{\ln x-\mu}{\sigma\sqrt2}\Big)\Big]f(x) = exp(−(ln x − μ)²/2σ²) / (xσ√2π)
X is log-normal if ln X is normal. Models positive, multiplicative quantities.
Worked example: mu = 0, s = 0.5, x0 = 1.5 → 0.38287 ✓ checked
See also: log-normal distribution: mean
f(x)=\frac{1}{x\sigma\sqrt{2\pi}}\exp\!\Big(-\frac{(\ln x-\mu)^2}{2\sigma^2}\Big),\ x>0E[X] = exp(μ + σ²/2)
Verified against numerical integration of the density.
Worked example: mu = 0, s = 0.5, x0 = 1.5 → 1.13315 ✓ checked
See also: log-normal distribution: variance
\mathbb E[X]=e^{\mu+\sigma^2/2}Var(X) = (e^σ² − 1) e^(2μ+σ²)
Verified against numerical integration of the density.
Worked example: mu = 0, s = 0.5, x0 = 1.5 → 0.364696 ✓ checked
See also: log-normal distribution: mean
\operatorname{Var}(X)=\big(e^{\sigma^2}-1\big)e^{2\mu+\sigma^2}F(x) = 1/(1 + e^(−(x−μ)/s))
Verified against numerical integration of the density up to x.
Worked example: mu = 1, s = 2, x0 = 0.5 → 0.437823 ✓ checked
See also: logistic distribution: density
F(x)=\frac{1}{1+e^{-(x-\mu)/s}}f(x) = e^(−z) / (s(1 + e^(−z))²), z = (x−μ)/s
Its cdf is the sigmoid, which is why logistic regression uses it.
Worked example: mu = 1, s = 2, x0 = 0.5 → 0.123067 ✓ checked
See also: logistic distribution: mean
f(x)=\frac{e^{-(x-\mu)/s}}{s\big(1+e^{-(x-\mu)/s}\big)^2}E[X] = μ
Verified against numerical integration of the density.
Worked example: mu = 1, s = 2, x0 = 0.5 → 1 ✓ checked
See also: logistic distribution: variance
\mathbb E[X]=\muVar(X) = s²π²/3
Verified against numerical integration of the density.
Worked example: mu = 1, s = 2, x0 = 0.5 → 13.1595 ✓ checked
See also: logistic distribution: mean
\operatorname{Var}(X)=\frac{s^2\pi^2}{3}(E|X|ʳ)^(1/r) ≤ (E|X|ˢ)^(1/s)
Higher moments dominate: E|X| ≤ √(E X²).
Worked example: x = [1, 3], p = [0.5, 0.5] → [2, 2.23607] ✓ checked
See also: Jensen's inequality
\big(\mathbb E|X|^r\big)^{1/r}\le\big(\mathbb E|X|^s\big)^{1/s}\quad(0<r<s)P(X ≥ a) ≤ E[X] / a
Loose, but needs only the mean.
Worked example: ex = 2, a = 5 → 0.4 ✓ checked
See also: Chebyshev's inequality · Hoeffding's inequality
P(X\ge a)\le \frac{\mathbb{E}[X]}{a}\quad (X\ge0,\ a>0)f(x) = √(2/π) x² e^(−x²/2a²) / a³
The speed of a particle whose three velocity components are independent N(0, a²).
Worked example: a = 1.5, x0 = 2 → 0.388765 ✓ checked
See also: Maxwell–Boltzmann distribution: mean
f(x)=\sqrt{\tfrac2\pi}\,\frac{x^2e^{-x^2/2a^2}}{a^3}E[X] = 2a√(2/π)
Verified against numerical integration of the density.
Worked example: a = 1.5, x0 = 2 → 2.39365 ✓ checked
See also: Maxwell–Boltzmann distribution: variance
\mathbb E[X]=2a\sqrt{\tfrac2\pi}Var(X) = a²(3π − 8)/π
Verified against numerical integration of the density.
Worked example: a = 1.5, x0 = 2 → 1.02042 ✓ checked
See also: Maxwell–Boltzmann distribution: mean
\operatorname{Var}(X)=\frac{a^2(3\pi-8)}{\pi}E[Xⁿ] = M⁽ⁿ⁾(0)
Differentiate the MGF n times and set t = 0.
Worked example: mu = 1, s = 2 → [1, 5] ✓ checked
See also: normal distribution: moment generating function · cumulant generating function
\mathbb E[X^n]=M_X^{(n)}(0)=\frac{d^n}{dt^n}M_X(t)\Big|_{t=0}P(x) = n!/(Π xₖ!) · Π pₖ^(xₖ)
Mean npₖ, variance npₖ(1−pₖ), covariance −npᵢpⱼ.
Worked example: x = [2, 1, 1], p = [0.5, 0.3, 0.2] → 0.18 ✓ checked
See also: binomial distribution · Dirichlet distribution · categorical distribution
P(\mathbf x)=\frac{n!}{\prod_k x_k!}\prod_k p_k^{x_k}f(x) = (2π)^(−k/2) |Σ|^(−1/2) exp( −½ (x−μ)ᵀ Σ⁻¹ (x−μ) )
where k = dimension, μ = mean vector, Σ = covariance matrix (positive definite)
Worked example: x = [0, 0], mu = [0, 0], S = [[1, 0], [0, 1]] → 0.159155 ✓ checked
See also: normal (Gaussian) density
f(\mathbf{x}) = (2\pi)^{-k/2}\,|\Sigma|^{-1/2}\exp\!\left(-\tfrac12(\mathbf{x}-\boldsymbol\mu)^{\top}\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu)\right)μ₁|₂ = μ₁ + Σ₁₂ Σ₂₂⁻¹ (x₂ − μ₂); Σ₁|₂ = Σ₁₁ − Σ₁₂ Σ₂₂⁻¹ Σ₂₁
Conditioning a Gaussian gives a Gaussian whose covariance does not depend on the observed value. The basis of Gaussian processes and Kalman filters.
Worked example: S = [[1, 0.8], [0.8, 1]], x2 = 1 → [0.8, 0.36] ✓ checked
See also: multivariate normal density · Gaussian process posterior
\boldsymbol\mu_{1|2}=\boldsymbol\mu_1+\Sigma_{12}\Sigma_{22}^{-1}(\mathbf x_2-\boldsymbol\mu_2),\quad \Sigma_{1|2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}x ~ N(μ, Σ) ⇒ Ax + b ~ N(Aμ + b, AΣAᵀ)
Marginals are sub-vectors of μ and sub-blocks of Σ.
Worked example: A = [[1, 1]], S = [[1, 0.5], [0.5, 2]] → 4 ✓ checked
See also: multivariate normal: conditional distribution
\mathbf x\sim\mathcal N(\boldsymbol\mu,\Sigma)\ \Rightarrow\ A\mathbf x+\mathbf b\sim\mathcal N\big(A\boldsymbol\mu+\mathbf b,\ A\Sigma A^{\top}\big)E[X] = r(1 − p)/p
Verified against summation of the mass function.
Worked example: r = 3, p = 0.4, x0 = 2 → 4.5 ✓ checked
See also: negative binomial distribution: variance
\mathbb E[X]=\frac{r(1-p)}{p}P(X = k) = C(k+r−1, k) (1−p)ᵏ pʳ
The number of failures before the r-th success. A gamma–Poisson mixture, so it models overdispersed counts.
Worked example: r = 3, p = 0.4, x0 = 2 → 0.13824 ✓ checked
See also: negative binomial distribution: mean
P(X=k)=\binom{k+r-1}{k}(1-p)^k p^rVar(X) = r(1 − p)/p²
Verified against summation of the mass function.
Worked example: r = 3, p = 0.4, x0 = 2 → 11.25 ✓ checked
See also: negative binomial distribution: mean
\operatorname{Var}(X)=\frac{r(1-p)}{p^2}f(x) = 1/(σ√2π) · exp( −(x−μ)² / 2σ² )
where μ = mean, σ = standard deviation
Maximum entropy distribution for a given mean and variance.
Worked example: x = 0, mu = 0, s = 1 → 0.398942 ✓ checked
See also: multivariate normal density · normal cumulative distribution · differential entropy of a Gaussian
f(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)P(X ≤ k) ≈ Φ((k + ½ − np)/√(np(1−p)))
The ½ is the continuity correction. Use when np and n(1−p) are both at least 10.
Worked example: n = 100, p = 0.5, k = 55 → 0.864334 ✓ checked
See also: binomial distribution: mean · normal cumulative distribution
\operatorname{Bin}(n,p)\approx\mathcal N\big(np,\ np(1-p)\big),\qquad P(X\le k)\approx\Phi\!\Big(\frac{k+\frac12-np}{\sqrt{np(1-p)}}\Big)Φ(x) = ½ [ 1 + erf(x/√2) ]
Φ(1.96) ≈ 0.975, the source of the 95% interval.
Worked example: x = 1.96 → 0.975002 ✓ checked
See also: normal (Gaussian) density · z-score
\Phi(x) = \tfrac12\left[1+\operatorname{erf}\!\left(\frac{x}{\sqrt2}\right)\right]φ(t) = exp(iμt − σ²t²/2)
The characteristic function always exists, even for the Cauchy.
Worked example: mu = 1, s = 2, t = 0.5 → 0.532281+0.290786j ✓ checked
See also: normal distribution: moment generating function
\varphi_X(t)=\mathbb E[e^{itX}]=\exp\!\big(i\mu t-\tfrac12\sigma^2t^2\big)E[X] = μ
Verified against numerical integration of the density.
Worked example: mu = 1, s = 2, x0 = 0 → 1 ✓ checked
See also: normal distribution: variance
\mathbb E[X]=\muM(t) = exp(μt + σ²t²/2)
Derivatives at 0 give the moments.
Worked example: mu = 1, s = 2, t = 0.3 → 1.61607 ✓ checked
See also: normal distribution: mean
M_X(t)=\mathbb E[e^{tX}]=\exp\!\big(\mu t+\tfrac12\sigma^2t^2\big)Var(X) = σ²
Verified against numerical integration of the density.
Worked example: mu = 1, s = 2, x0 = 0 → 4 ✓ checked
See also: normal distribution: mean
\operatorname{Var}(X)=\sigma^2mean-reverting; stationary variance σ²/(2θ)
The continuous-time AR(1); also the forward process of the variance-preserving diffusion model.
Worked example: theta = 0.5, mu = 1, x0 = 3, t = 2, s = 0.4 → [1.73576, 0.16] ✓ checked
See also: AR(1) process · Brownian motion (Wiener process)
dX=\theta(\mu-X)\,dt+\sigma\,dW,\quad \mathbb E X_t=\mu+(x_0-\mu)e^{-\theta t},\quad \operatorname{Var}(X_\infty)=\frac{\sigma^2}{2\theta}F(x) = 1 − (x_m/x)ᵅ
Verified against numerical integration of the density up to x.
Worked example: xm = 1, al = 4, x0 = 2 → 0.9375 ✓ checked
See also: Pareto (type I) distribution: density
F(x)=1-\Big(\frac{x_m}{x}\Big)^{\alpha}f(x) = α x_mᵅ / x^(α+1) for x ≥ x_m
Heavy-tailed power law: the 80/20 rule.
Worked example: xm = 1, al = 4, x0 = 2 → 0.125 ✓ checked
See also: Pareto (type I) distribution: mean
f(x)=\frac{\alpha x_m^{\alpha}}{x^{\alpha+1}},\ x\ge x_mE[X] = α x_m/(α − 1)
Verified against numerical integration of the density.
Worked example: xm = 1, al = 4, x0 = 2 → 1.33337 ✓ checked
See also: Pareto (type I) distribution: variance
\mathbb E[X]=\frac{\alpha x_m}{\alpha-1}\ \ (\alpha>1)Var(X) = x_m² α / ((α−1)²(α−2))
Verified against numerical integration of the density.
Worked example: xm = 1, al = 4, x0 = 2 → 0.222232 ✓ checked
See also: Pareto (type I) distribution: mean
\operatorname{Var}(X)=\frac{x_m^2\alpha}{(\alpha-1)^2(\alpha-2)}\ \ (\alpha>2)top p of the population holds p^(1 − 1/α) of the total
α ≈ 1.16 gives the 80/20 rule.
Worked example: p = 0.2, al = 1.16 → 0.800922 ✓ checked
See also: Pareto (type I) distribution: density · Gini coefficient and Lorenz curve
\text{share of the total held by the top fraction }p=p^{\,1-1/\alpha}Bin(n, p) ≈ Poisson(np)
Good when n ≥ 20 and p ≤ 0.05.
Worked example: n = 200, p = 0.01, k = 2 → [0.272033, 0.270671] ✓ checked
See also: binomial distribution · Poisson distribution
\binom nk p^k(1-p)^{n-k}\ \approx\ \frac{(np)^ke^{-np}}{k!}\quad(n\text{ large},\ p\text{ small})P(X = k) = λᵏ e^(−λ) / k!
where λ = rate (the mean and the variance)
Limit of Binomial(n, λ/n) as n → ∞.
Worked example: lam = 2, k = 3 → 0.180447 ✓ checked
See also: binomial distribution · exponential distribution
P(X=k) = \frac{\lambda^{k}e^{-\lambda}}{k!}E[X] = λ
Verified against summation of the mass function.
Worked example: lam = 3.5, x0 = 3 → 3.5 ✓ checked
See also: Poisson distribution: variance
\mathbb E[X]=\lambdaM(t) = exp(λ(eᵗ − 1))
Worked example: lam = 3.5, t = 0.3 → 3.40248 ✓ checked
See also: Poisson distribution: mean
M_X(t)=\exp\!\big(\lambda(e^{t}-1)\big)Var(X) = λ
Verified against summation of the mass function.
Worked example: lam = 3.5, x0 = 3 → 3.5 ✓ checked
See also: Poisson distribution: mean
\operatorname{Var}(X)=\lambdaN_t ~ Poisson(λt); waiting times ~ Exp(λ)
Worked example: lam = 3, t = 2, k = 4 → 0.133853 ✓ checked
See also: Poisson distribution · exponential distribution
N_t\sim\mathrm{Poisson}(\lambda t),\qquad T_i\sim\mathrm{Exp}(\lambda)\ \text{(inter-arrival times)},\qquad \mathbb E[T_i]=\frac1\lambdaF(x) = 1 − e^(−x²/2σ²)
Verified against numerical integration of the density up to x.
Worked example: s = 2, x0 = 1.5 → 0.24516 ✓ checked
See also: Rayleigh distribution: density
F(x)=1-e^{-x^2/2\sigma^2}f(x) = (x/σ²) e^(−x²/2σ²)
The length of a 2-D vector with independent N(0, σ²) components.
Worked example: s = 2, x0 = 1.5 → 0.283065 ✓ checked
See also: Rayleigh distribution: mean
f(x)=\frac{x}{\sigma^2}e^{-x^2/2\sigma^2},\ x\ge0E[X] = σ√(π/2)
Verified against numerical integration of the density.
Worked example: s = 2, x0 = 1.5 → 2.50663 ✓ checked
See also: Rayleigh distribution: variance
\mathbb E[X]=\sigma\sqrt{\tfrac\pi2}Var(X) = (4 − π)σ²/2
Verified against numerical integration of the density.
Worked example: s = 2, x0 = 1.5 → 1.71681 ✓ checked
See also: Rayleigh distribution: mean
\operatorname{Var}(X)=\frac{4-\pi}{2}\sigma^2renewals per unit time → 1/μ
where μ = mean inter-renewal time
Worked example: mu = 4, t = 100 → 25 ✓ checked
See also: Poisson process
\frac{N(t)}t\to\frac1\mu\ \text{a.s.},\qquad \mathbb E[N(t)]\approx\frac t\mux̄ ~ N(μ, σ²/n); (n − 1)s²/σ² ~ χ²ₙ₋₁; t and F ratios
Worked example: n = 10, s2 = 4 → [0.4, 18] ✓ checked
See also: chi-square distribution: mean · Student's t distribution: density · F distribution distribution: density
\bar X\sim\mathcal N\Big(\mu,\frac{\sigma^2}n\Big),\quad \frac{(n-1)s^2}{\sigma^2}\sim\chi^2_{n-1},\quad \frac{\bar X-\mu}{s/\sqrt n}\sim t_{n-1},\quad \frac{s_1^2/\sigma_1^2}{s_2^2/\sigma_2^2}\sim F_{n_1-1,n_2-1}t_mix ≤ (1/γ) ln(1/(ε π_min))
where γ = spectral gap
A larger gap means faster convergence to the stationary distribution.
Worked example: lam2 = 0.4, eps = 0.01, pimin = 0.166667 → 10.6615 ✓ checked
See also: stationary distribution of a Markov chain · Gelman–Rubin R̂
t_{\text{mix}}(\varepsilon)\le\frac1{\gamma}\ln\frac1{\varepsilon\,\pi_{\min}},\qquad \gamma=1-\lambda_2f(x) = Γ((ν+1)/2) / (√(νπ) Γ(ν/2)) · (1 + x²/ν)^(−(ν+1)/2)
Heavier tails than the normal; tends to the normal as ν grows. Used for small-sample inference.
Worked example: nu = 5, x0 = 1.2 → 0.177659 ✓ checked
See also: Student's t distribution: mean
f(x)=\frac{\Gamma\!\big(\frac{\nu+1}{2}\big)}{\sqrt{\nu\pi}\,\Gamma\!\big(\frac\nu2\big)}\Big(1+\frac{x^2}{\nu}\Big)^{-\frac{\nu+1}{2}}E[X] = 0 (ν > 1)
Verified against numerical integration of the density.
Worked example: nu = 5, x0 = 1.2 → 1.11022e-16 ✓ checked
See also: Student's t distribution: variance
\mathbb E[X]=0\ \ (\nu>1)Var(X) = ν/(ν−2) (ν > 2)
Verified against numerical integration of the density.
Worked example: nu = 5, x0 = 1.2 → 1.66667 ✓ checked
See also: Student's t distribution: mean
\operatorname{Var}(X)=\frac{\nu}{\nu-2}\ \ (\nu>2)Σ aᵢXᵢ ~ N(Σ aᵢμᵢ, Σ aᵢ²σᵢ²)
Variances add, standard deviations do not.
Worked example: a = [1, 2], mu = [1, 3], v = [4, 1] → [7, 8] ✓ checked
See also: normal (Gaussian) density
X_i\sim\mathcal N(\mu_i,\sigma_i^2)\ \Rightarrow\ \sum_i a_iX_i\sim\mathcal N\Big(\sum_i a_i\mu_i,\ \sum_i a_i^2\sigma_i^2\Big)f(x) = 2(x−a)/((b−a)(c−a)) for x < c; 2(b−x)/((b−a)(b−c)) for x ≥ c
Min a, max b, mode c: a quick model when only those three are known.
Worked example: a = 0, b = 4, c = 1, x0 = 2 → 0.333333 ✓ checked
See also: triangular distribution: mean
f(x)=\begin{cases}\frac{2(x-a)}{(b-a)(c-a)}&a\le x<c\\[2pt]\frac{2(b-x)}{(b-a)(b-c)}&c\le x\le b\end{cases}E[X] = (a + b + c)/3
Verified against numerical integration of the density.
Worked example: a = 0, b = 4, c = 1, x0 = 2 → 1.66667 ✓ checked
See also: triangular distribution: variance
\mathbb E[X]=\frac{a+b+c}{3}Var(X) = (a²+b²+c²−ab−ac−bc)/18
Verified against numerical integration of the density.
Worked example: a = 0, b = 4, c = 1, x0 = 2 → 0.722222 ✓ checked
See also: triangular distribution: mean
\operatorname{Var}(X)=\frac{a^2+b^2+c^2-ab-ac-bc}{18}M(t) = (e^(tb) − e^(ta)) / (t(b − a))
Worked example: a = 1, b = 4, t = 0.7 → 6.87185 ✓ checked
See also: continuous uniform distribution: mean
M_X(t)=\frac{e^{tb}-e^{ta}}{t\,(b-a)}\quad(t\ne0)P(∪Aᵢ) ≤ ΣP(Aᵢ)
The workhorse of learning-theory proofs.
Worked example: p = [0.1, 0.2, 0.05] → 0.35 ✓ checked
See also: inclusion–exclusion · generalisation bound for a finite class
P\Big(\bigcup_iA_i\Big)\le\sum_iP(A_i),\qquad \sum_iP(A_i)-\sum_{i<j}P(A_i\cap A_j)\le P\Big(\bigcup_iA_i\Big)Var(X) = E[X²] − E[X]² σ = √Var(X)
where σ = standard deviation
Var(aX + b) = a² Var(X). Variances of independent variables add.
Worked example: x = [1, 2, 3], p = [0.2, 0.5, 0.3] → 0.49 ✓ checked
See also: expected value · covariance · sample variance (Bessel's correction)
\operatorname{Var}(X) = \mathbb{E}[X^2] - \mathbb{E}[X]^2 \qquad \sigma = \sqrt{\operatorname{Var}(X)}Var(aX + bY) = a²Var X + b²Var Y + 2ab Cov(X, Y)
Worked example: a = 2, b = -1, vx = 4, vy = 9, c = 3 → 13 ✓ checked
See also: covariance · variance and standard deviation
\operatorname{Var}(aX+bY)=a^2\operatorname{Var}X+b^2\operatorname{Var}Y+2ab\operatorname{Cov}(X,Y),\qquad \operatorname{Var}\Big(\sum_iX_i\Big)=\sum_i\operatorname{Var}X_i\ \text{(independent)}E[S_N] = E[N]·E[X]
where N a stopping time
Worked example: EN = 8, EX = 2.5 → 20 ✓ checked
See also: compound Poisson process
\mathbb E\Big[\sum_{i=1}^{N}X_i\Big]=\mathbb E[N]\,\mathbb E[X]F(x) = 1 − e^(−(x/λ)ᵏ)
Verified against numerical integration of the density up to x.
Worked example: k = 1.5, lam = 2, x0 = 1.5 → 0.477703 ✓ checked
See also: Weibull distribution: density
F(x)=1-e^{-(x/\lambda)^k}f(x) = (k/λ)(x/λ)^(k−1) e^(−(x/λ)ᵏ)
Failure times; k < 1 decreasing hazard, k > 1 increasing hazard, k = 1 exponential.
Worked example: k = 1.5, lam = 2, x0 = 1.5 → 0.339242 ✓ checked
See also: Weibull distribution: mean
f(x)=\frac k\lambda\Big(\frac x\lambda\Big)^{k-1}e^{-(x/\lambda)^k}E[X] = λ Γ(1 + 1/k)
Verified against numerical integration of the density.
Worked example: k = 1.5, lam = 2, x0 = 1.5 → 1.80549 ✓ checked
See also: Weibull distribution: variance
\mathbb E[X]=\lambda\,\Gamma\!\big(1+\tfrac1k\big)Var(X) = λ²[Γ(1 + 2/k) − Γ(1 + 1/k)²]
Verified against numerical integration of the density.
Worked example: k = 1.5, lam = 2, x0 = 1.5 → 1.50276 ✓ checked
See also: Weibull distribution: mean
\operatorname{Var}(X)=\lambda^2\Big[\Gamma\!\big(1+\tfrac2k\big)-\Gamma\!\big(1+\tfrac1k\big)^2\Big]frequency of the k-th most common item ∝ 1/k^s
Word frequencies follow it with s ≈ 1.
Worked example: N = 1000, s = 1, k = 1 → 0.133592 ✓ checked
See also: Pareto (type I) distribution: density
f(k)=\frac{k^{-s}}{\sum_{n=1}^{N}n^{-s}},\qquad f(k)\propto\frac1{k^{s}}p̂ₖ = (cₖ + α)/(N + αK)
where α = 1 is Laplace's rule of succession
Avoids zero probabilities for unseen events (naive Bayes, n-gram models).
Worked example: c = 0, a = 1, N = 10, K = 5 → 0.0666667 ✓ checked
See also: Dirichlet–multinomial conjugate update
\hat p_k=\frac{c_k+\alpha}{N+\alpha K}AIC = 2k − 2 ln L̂
where k = parameters, L̂ = maximised likelihood
Lower is better.
Worked example: k = 3, lnl = -100 → 206 ✓ checked
See also: Bayesian information criterion
\mathrm{AIC}=2k-2\ln\hat Lxₜ = φxₜ₋₁ + εₜ; Var = σ²/(1 − φ²); ρₖ = φᵏ
Stationary only if |φ| < 1; φ = 1 is a random walk.
Worked example: phi = 0.8, s2 = 1, k = 3 → [2.77778, 0.512] ✓ checked
See also: random walk · Yule–Walker equations (AR(2))
x_t=\phi x_{t-1}+\varepsilon_t,\quad \operatorname{Var}(x_t)=\frac{\sigma^2}{1-\phi^2},\quad \rho_k=\phi^k\quad(|\phi|<1)σₜ² = ω + αεₜ₋₁² + βσₜ₋₁²
where β = 0 gives ARCH(1)
Stationary variance ω/(1 − α − β).
Worked example: w = 0.1, a = 0.1, be = 0.8, e = 2, s2 = 1 → [1.3, 1] ✓ checked
See also: EWMA volatility (RiskMetrics)
\sigma_t^2=\omega+\alpha\varepsilon_{t-1}^2+\beta\sigma_{t-1}^2ρ̂ₖ = Σ (xₜ − x̄)(xₜ₊ₖ − x̄) / Σ (xₜ − x̄)²
where k = lag
Worked example: x = [1, 2, 3, 4, 5], k = 1 → 0.4 ✓ checked
See also: Durbin–Watson statistic · Ljung–Box statistic
\hat\rho_k=\frac{\sum_{t=1}^{n-k}(x_t-\bar x)(x_{t+k}-\bar x)}{\sum_{t=1}^{n}(x_t-\bar x)^2}ATE = E[Y(1) − Y(0)]; difference in means when T is randomised
Without randomisation, confounding makes the difference in means biased.
Worked example: t = [10, 12, 14], c = [8, 9, 10] → 3 ✓ checked
See also: inverse-propensity weighting · backdoor adjustment
\mathrm{ATE}=\mathbb E[Y(1)-Y(0)]=\mathbb E[Y\mid T=1]-\mathbb E[Y\mid T=0]\ \ (\text{randomised})P(y | do(x)) = Σ_z P(y | x, z) P(z)
where Z blocks all backdoor paths from X to Y
Pearl's adjustment formula; conditioning on a collider or mediator would break it.
Worked example: pyxz = [0.8, 0.4], pz = [0.3, 0.7] → 0.52 ✓ checked
See also: average treatment effect · Bayes' theorem
P\big(y\mid\operatorname{do}(x)\big)=\sum_zP(y\mid x,z)\,P(z)posterior odds = Bayes factor × prior odds
Above 10 is usually called strong evidence for H₁.
Worked example: bf = 6, prior = 0.5 → 6 ✓ checked
See also: Bayes' theorem
\mathrm{BF}_{10}=\frac{p(D\mid H_1)}{p(D\mid H_0)},\qquad \frac{P(H_1\mid D)}{P(H_0\mid D)}=\mathrm{BF}_{10}\cdot\frac{P(H_1)}{P(H_0)}compare posterior samples of each arm's conversion rate
Worked example: sa = 100, fa = 900, sb = 120, fb = 880 → [0.100798, 0.120758] ✓ checked
See also: Beta–binomial conjugate update
p_A\sim\mathrm{Beta}(1+s_A,1+f_A),\quad P(p_B>p_A)\approx\frac1M\sum_m\mathbf 1\big[p_B^{(m)}>p_A^{(m)}\big]BIC = k ln n − 2 ln L̂
where n = number of observations
Penalises complexity more than AIC when n > 7.
Worked example: k = 3, n = 100, lnl = -100 → 213.816 ✓ checked
See also: Akaike information criterion
\mathrm{BIC}=k\ln n-2\ln\hat Lk = max{i : p₍ᵢ₎ ≤ (i/m)q}; reject the k smallestwhere q = target false discovery rate
Controls the expected share of false discoveries, not the chance of any.
Worked example: p = [0.001, 0.012, 0.03, 0.2], q = 0.05 → 3 ✓ checked
See also: Holm–Bonferroni procedure · Bonferroni correction
\text{reject all } p_{(i)}\text{ with } i\le k,\quad k=\max\Big\{i:\ p_{(i)}\le\frac{i}{m}q\Big\}P(Σ(Xᵢ − EXᵢ) ≥ t) ≤ exp(−(t²/2)/(ΣVar + Mt/3))
where |Xᵢ − EXᵢ| ≤ M
Tighter than Hoeffding when the variance is small.
Worked example: t = 10, v = 25, M = 1 → 0.171237 ✓ checked
See also: Hoeffding's inequality
P\Big(\sum_i(X_i-\mathbb E X_i)\ge t\Big)\le\exp\!\Big(-\frac{t^2/2}{\sum_i\operatorname{Var}(X_i)+Mt/3}\Big)sup|Fₙ − Φ| ≤ Cρ/(σ³√n)
A quantitative CLT: the normal approximation error shrinks like 1/√n.
Worked example: C = 0.4748, rho = 2, s = 1, n = 100 → 0.09496 ✓ checked
See also: central limit theorem (standardised mean)
\sup_x\big|F_n(x)-\Phi(x)\big|\le\frac{C\,\rho}{\sigma^3\sqrt n},\quad \rho=\mathbb E|X-\mu|^3,\ C\le0.4748posterior = Beta(α + x, β + n − x)
Posterior mean (α + x)/(α + β + n): a weighted average of prior mean and sample proportion.
Worked example: al = 2, be = 2, x = 7, n = 10 → [9, 5, 0.642857] ✓ checked
See also: beta distribution: mean · maximum a posteriori (Gaussian mean, Gaussian prior) · additive (Laplace) smoothing
\theta\sim\mathrm{Beta}(\alpha,\beta),\ x\text{ successes in }n\ \Rightarrow\ \theta\mid x\sim\mathrm{Beta}(\alpha+x,\ \beta+n-x)MSE = Var + Bias²
A little bias can buy a lot of variance reduction (ridge, James–Stein).
Worked example: var = 0.04, bias = 0.1 → 0.05 ✓ checked
See also: bias–variance decomposition · James–Stein estimator
\operatorname{Bias}(\hat\theta)=\mathbb E\hat\theta-\theta,\qquad \mathrm{MSE}(\hat\theta)=\operatorname{Var}(\hat\theta)+\operatorname{Bias}(\hat\theta)^2E[(y − f̂)²] = Bias² + Variance + σ²
where σ² = irreducible noise
Flexible models trade bias for variance.
Worked example: bias = 0.5, var = 0.3, noise = 0.2 → 0.75 ✓ checked
See also: mean squared error
\mathbb{E}\big[(y-\hat f(x))^2\big]=\operatorname{Bias}[\hat f]^2+\operatorname{Var}[\hat f]+\sigma^2d̄ ± 1.96 s_d
where d = difference between two measurement methods
Worked example: dbar = 0.5, sd = 2 → [-3.42, 4.42] ✓ checked
See also: Cohen's kappa
\bar d\pm1.96\,s_dα_each = α / m
where m = number of tests
Controls the family-wise error rate; conservative.
Worked example: alpha = 0.05, m = 10 → 0.005 ✓ checked
See also: one-sample t statistic
\alpha_{\text{each}}=\frac{\alpha}{m}SE_boot = std of the statistic over B resamples
where θ̂*_b = statistic on resample b
Resample the data with replacement B times.
Worked example: t = [1, 2, 3, 4] → 1.29099 ✓ checked
See also: jackknife variance estimate
\widehat{\mathrm{SE}}_{\text{boot}}=\sqrt{\frac1{B-1}\sum_{b=1}^{B}\big(\hat\theta^{*}_b-\bar{\hat\theta}^{*}\big)^2}(yᵏ − 1)/λ, or ln y when λ = 0
where y > 0
Stabilises variance and makes data more normal.
Worked example: y = 4, lam = 0.5 → 2 ✓ checked
See also: logarithm and exponent rules
y^{(\lambda)}=\begin{cases}\dfrac{y^\lambda-1}{\lambda}&\lambda\ne0\\[2pt]\ln y&\lambda=0\end{cases}P(X − μ ≥ t) ≤ σ²/(σ² + t²)
The one-sided Chebyshev.
Worked example: s2 = 4, t = 4 → 0.2 ✓ checked
See also: Chebyshev's inequality
P(X-\mu\ge t)\le\frac{\sigma^2}{\sigma^2+t^2}Zₙ = √n (X̄ₙ − μ) / σ → N(0, 1)
where X̄ₙ = sample mean, μ and σ = true mean and standard deviation
Needs i.i.d. samples with finite variance.
Worked example: n = 100, xbar = 10.5, mu = 10, s = 2 → 2.5 ✓ checked
See also: standard error of the mean · z-score
Z_n=\frac{\sqrt{n}\,(\bar X_n-\mu)}{\sigma}\ \xrightarrow{d}\ \mathcal N(0,1)P(X ≥ (1 + δ)μ) ≤ exp(−δ²μ/(2 + δ))
where X a sum of independent Bernoulli variables, μ = E[X]
Worked example: mu = 50, d = 0.2 → 0.40289 ✓ checked
See also: Hoeffding's inequality · Markov's inequality
P(X\ge(1+\delta)\mu)\le\exp\!\Big(-\frac{\delta^2\mu}{2+\delta}\Big)χ² = Σ (Oᵢ − Eᵢ)²/Eᵢ
where O observed counts, E expected counts, m fitted parameters
Needs expected counts of about 5 or more.
Worked example: O = [18, 22, 20, 40], E = [25, 25, 25, 25] → 12.32 ✓ checked
See also: chi-square test of independence
\chi^2=\sum_i\frac{(O_i-E_i)^2}{E_i},\quad \nu=k-1-mEᵢⱼ = RᵢCⱼ/N; χ² = Σ (Oᵢⱼ − Eᵢⱼ)²/Eᵢⱼ
where R, C = row and column totals
Worked example: O = [[20, 30], [30, 20]] → 4 ✓ checked
See also: chi-square goodness of fit · Cramér's V
E_{ij}=\frac{R_iC_j}{N},\qquad \chi^2=\sum_{ij}\frac{(O_{ij}-E_{ij})^2}{E_{ij}},\quad \nu=(r-1)(c-1)CV = σ/μ
Unit-free spread; meaningful only for positive ratio-scale data.
Worked example: s = 2, m = 10 → 0.2 ✓ checked
See also: variance and standard deviation
\mathrm{CV}=\frac{\sigma}{\mu}d = (μ₁ − μ₂) / s_pooled
About 0.2 small, 0.5 medium, 0.8 large (rules of thumb).
Worked example: m1 = 10, m2 = 8, s = 4 → 0.5 ✓ checked
See also: one-sample t statistic
d=\frac{\mu_1-\mu_2}{s_{\text{pooled}}}κ = (p_o − p_e)/(1 − p_e)
where p_o = observed agreement, p_e = agreement expected by chance
0 is chance-level agreement, 1 is perfect.
Worked example: po = 0.85, pe = 0.5 → 0.7 ✓ checked
See also: intraclass correlation (one-way)
\kappa=\frac{p_o-p_e}{1-p_e}(x̄₁ − x̄₂) ± t·√(s₁²/n₁ + s₂²/n₂)
Worked example: d = 2, t = 2, se2 = 0.5 → [0.585786, 3.41421] ✓ checked
See also: Welch's t test
(\bar x_1-\bar x_2)\pm t_{\nu,1-\alpha/2}\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}x̄ ± z · s/√n
where z = 1.96 for 95%; use t instead of z for small n
Frequentist: 95% of such intervals contain the true mean.
Worked example: xbar = 10, s = 2, n = 100, z = 1.96 → [9.608, 10.392] ✓ checked
See also: standard error of the mean · one-sample t statistic
\bar x \pm z_{1-\alpha/2}\,\frac{s}{\sqrt n}((n−1)s²/χ²_upper, (n−1)s²/χ²_lower)
where χ² quantiles with n − 1 degrees of freedom
Sensitive to non-normality.
Worked example: n = 20, s2 = 4, qlo = 8.907, qhi = 32.852 → [2.31341, 8.53261] ✓ checked
See also: sample variance (Bessel's correction)
\Big(\frac{(n-1)s^2}{\chi^2_{1-\alpha/2,\,n-1}},\ \frac{(n-1)s^2}{\chi^2_{\alpha/2,\,n-1}}\Big)exp(ln OR ± z√(1/a + 1/b + 1/c + 1/d))
Worked example: a = 20, b = 80, c = 10, d = 90, z = 1.96 → [0.99428, 5.09162] ✓ checked
See also: odds ratio and relative risk
\exp\Big(\ln\widehat{\mathrm{OR}}\pm z_{1-\alpha/2}\sqrt{\tfrac1a+\tfrac1b+\tfrac1c+\tfrac1d}\Big)Dᵢ = eᵢ²/(p s²) · hᵢᵢ/(1 − hᵢᵢ)²
where eᵢ = residual, p = parameters
Large values flag influential points (rule of thumb: > 4/n).
Worked example: e = 2, p = 2, s2 = 1, h = 0.4 → 2.22222 ✓ checked
See also: hat matrix and leverage
D_i=\frac{e_i^2}{p\,s^2}\cdot\frac{h_{ii}}{(1-h_{ii})^2}AICc = AIC + 2k(k + 1)/(n − k − 1)
Use when n/k is below about 40.
Worked example: aic = 206, k = 3, n = 20 → 207.5 ✓ checked
See also: Akaike information criterion · Bayesian information criterion
\mathrm{AIC}_c=\mathrm{AIC}+\frac{2k(k+1)}{n-k-1}Var(β̂) = σ²(XᵀX)⁻¹
OLS is the best linear unbiased estimator when errors are uncorrelated with constant variance.
Worked example: X = [[1, 0], [1, 1], [1, 2]], s2 = 2 → 1 ✓ checked
See also: ordinary least squares (normal equations) · standard error of the regression slope
\operatorname{Var}(\hat\beta)=\sigma^2(X^{\top}X)^{-1}h(t|x) = h₀(t) exp(βᵀx)
where h₀ = baseline hazard
exp(β) is the hazard ratio for a one-unit increase in x.
Worked example: beta = 0.7 → 2.01375 ✓ checked
See also: hazard and survival functions
h(t\mid\mathbf x)=h_0(t)\,\exp\big(\boldsymbol\beta^{\top}\mathbf x\big)V = √( (χ²/N) / min(r−1, c−1) )
Effect size for association between categorical variables, from 0 to 1.
Worked example: chi2 = 4, N = 100, r = 2, c = 2 → 0.2 ✓ checked
See also: chi-square test of independence · phi coefficient (2×2)
V=\sqrt{\frac{\chi^2/N}{\min(r-1,\,c-1)}}Var(θ̂) ≥ 1 / (n·I(θ))
where for any unbiased estimator
Worked example: n = 100, info = 4 → 0.0025 ✓ checked
See also: Fisher information (Bernoulli)
\operatorname{Var}(\hat\theta)\ge\frac{1}{n\,I(\theta)}α = (k/(k−1))(1 − Σσᵢ²/σ_T²)
where k items, σ_T² = variance of the total score
A reliability measure of internal consistency.
Worked example: k = 4, vi = [1, 1, 1, 1], vt = 10 → 0.8 ✓ checked
See also: intraclass correlation (one-way)
\alpha=\frac{k}{k-1}\Big(1-\frac{\sum_i\sigma_i^2}{\sigma_T^2}\Big)adjust the metric with a pre-experiment covariate
A correlation of 0.7 halves the variance.
Worked example: rho = 0.7 → 0.51 ✓ checked
See also: control variates
Y^{*}=Y-\theta\,(X-\mathbb E X),\quad \theta=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}X},\quad \operatorname{Var}(Y^{*})=(1-\rho^2)\operatorname{Var}(Y)Sₜ = max(0, Sₜ₋₁ + xₜ − μ₀ − k); alarm when Sₜ > h
where k = allowance, h = threshold
Detects a small sustained shift faster than a Shewhart chart.
Worked example: x = [10, 10.5, 11, 12], mu0 = 10, k = 0.25 → 2.75 ✓ checked
See also: sequential probability ratio test
S_t=\max\big(0,\ S_{t-1}+x_t-\mu_0-k\big),\qquad \text{alarm if }S_t>hVar(g(θ̂)) ≈ g′(θ)² Var(θ̂)
Propagates uncertainty through a smooth function.
Worked example: gp = 2, v = 0.25 → 1 ✓ checked
See also: standard error of the mean
\sqrt n\big(g(\hat\theta)-g(\theta)\big)\xrightarrow{d}\mathcal N\big(0,\ g'(\theta)^2\sigma^2\big)deff = 1 + (m − 1)ρ
where m = cluster size, ρ = intraclass correlation
Correlated observations carry less information than independent ones.
Worked example: m = 20, rho = 0.05, n = 1000 → [1.95, 512.821] ✓ checked
See also: intraclass correlation (one-way)
\mathrm{deff}=1+(m-1)\rho,\qquad n_{\text{eff}}=\frac{n}{\mathrm{deff}}D = 2(ℓ_saturated − ℓ_model)
For the normal it is the residual sum of squares; compare nested models by the drop in deviance (≈ χ²).
Worked example: ls = -10, lm = -14 → 8 ✓ checked
See also: likelihood ratio test
D=2\big(\ell_{\text{saturated}}-\ell_{\text{model}}\big)τ̂ = (treated: post − pre) − (control: post − pre)
Needs parallel trends.
Worked example: tp = 12, tq = 10, cp = 9, cq = 8 → 1 ✓ checked
See also: average treatment effect
\hat\tau=\big(\bar Y_{T,\text{post}}-\bar Y_{T,\text{pre}}\big)-\big(\bar Y_{C,\text{post}}-\bar Y_{C,\text{pre}}\big)posterior = Dir(α + x)
Posterior mean (αₖ + xₖ)/(α₀ + n): additive smoothing.
Worked example: al = [1, 1, 1], x = [5, 3, 2] → [0.461538, 0.307692, 0.230769] ✓ checked
See also: additive (Laplace) smoothing
\mathbf p\sim\mathrm{Dir}(\boldsymbol\alpha),\ \text{counts }\mathbf x\ \Rightarrow\ \mathbf p\mid\mathbf x\sim\mathrm{Dir}(\boldsymbol\alpha+\mathbf x)d = Σ(eₜ − eₜ₋₁)²/Σeₜ² ≈ 2(1 − ρ̂₁)
Near 2 means no first-order autocorrelation.
Worked example: e = [1, -1, 1, -1] → 3 ✓ checked
See also: autocorrelation function
d=\frac{\sum_{t=2}^{n}(e_t-e_{t-1})^2}{\sum_{t=1}^{n}e_t^2}\approx2(1-\hat\rho_1)P(sup|Fₙ − F| > ε) ≤ 2 exp(−2nε²)
Gives a distribution-free confidence band for a cdf.
Worked example: n = 100, e = 0.1 → 0.270671 ✓ checked
See also: empirical distribution function
P\Big(\sup_x|F_n(x)-F(x)|>\varepsilon\Big)\le 2e^{-2n\varepsilon^2}df(λ) = Σ dⱼ²/(dⱼ² + λ)
where dⱼ = singular values of X
Falls from p (λ = 0) toward 0 as λ grows.
Worked example: d = [3, 1], lam = 1 → 1.4 ✓ checked
See also: ridge regression (closed form)
\mathrm{df}(\lambda)=\operatorname{tr}\big[X(X^{\top}X+\lambda I)^{-1}X^{\top}\big]=\sum_j\frac{d_j^2}{d_j^2+\lambda}Fₙ(x) = (1/n) Σ 1[xᵢ ≤ x]
Converges uniformly to F (Glivenko–Cantelli); DKW: P(sup|Fₙ − F| > ε) ≤ 2e^(−2nε²).
Worked example: x = [1, 2, 2, 5], t = 2 → 0.75 ✓ checked
See also: Kolmogorov–Smirnov statistic · Dvoretzky–Kiefer–Wolfowitz inequality
F_n(x)=\frac1n\sum_{i=1}^{n}\mathbf 1[x_i\le x]η² = SS_B / SS_T
The share of variance explained by group membership.
Worked example: ssb = 14, sst = 20 → 0.7 ✓ checked
See also: one-way ANOVA F statistic · Cohen's d (effect size)
\eta^2=\frac{\mathrm{SS}_B}{\mathrm{SS}_T}σₜ² = λσₜ₋₁² + (1 − λ)rₜ₋₁²
where λ = 0.94 for daily data
Worked example: lam = 0.94, s2 = 0.0001, r = 0.02 → 0.000118 ✓ checked
See also: ARCH(1) and GARCH(1,1) variance · simple exponential smoothing
\sigma_t^2=\lambda\sigma_{t-1}^2+(1-\lambda)r_{t-1}^2fpc = √((N − n)/(N − 1))
where N = population size
Matters once the sample is more than about 5% of the population.
Worked example: N = 1000, n = 100 → 0.949158 ✓ checked
See also: sample size to estimate a proportion · hypergeometric distribution: variance
\mathrm{fpc}=\sqrt{\frac{N-n}{N-1}},\qquad n_{\mathrm{adj}}=\frac{n}{1+\frac{n-1}N}I(p) = E[(∂ ln f/∂p)²] = 1 / (p(1−p))
Larger information means a sharper likelihood and a more precise estimate.
Worked example: p = 0.5 → 4 ✓ checked
See also: Cramér–Rao bound
I(p)=\mathbb{E}\!\left[\left(\frac{\partial}{\partial p}\ln f(X;p)\right)^{2}\right]=\frac{1}{p(1-p)}z = ½ ln((1 + r)/(1 − r)); SE = 1/√(n − 3)
Makes r approximately normal; build the interval on z and transform back with tanh.
Worked example: r = 0.6, n = 28 → [0.693147, 0.2] ✓ checked
See also: Pearson correlation
z=\operatorname{artanh}(r)=\tfrac12\ln\frac{1+r}{1-r},\qquad \mathrm{SE}=\frac1{\sqrt{n-3}}p = C(a+b, a) C(c+d, c) / C(n, a+c)
where table [[a, b], [c, d]], n = a+b+c+d
The hypergeometric probability of this exact table given the margins; sum over equal or more extreme tables for the p-value.
Worked example: a = 3, b = 1, c = 1, d = 3 → 0.228571 ✓ checked
See also: hypergeometric distribution: probability mass
p=\frac{\binom{a+b}{a}\binom{c+d}{c}}{\binom{n}{a+c}}X² = −2 Σ ln pᵢ ~ χ²(2k)
Worked example: p = [0.05, 0.1] → 10.5966 ✓ checked
See also: Stouffer's Z method
X^2=-2\sum_{i=1}^{k}\ln p_i\ \sim\ \chi^2_{2k}partial out the controls from both x and y, then regress the residuals
Explains what 'controlling for' means.
Worked example: x = [1, 2, 3, 4], z = [1, 1, 2, 2], y = [2, 3, 6, 7] → 2.66454e-15 ✓ checked
See also: ordinary least squares (normal equations) · orthogonal projection matrix
\hat\beta_1=\frac{\tilde x^{\top}\tilde y}{\tilde x^{\top}\tilde x},\quad \tilde x=M_2x,\ \tilde y=M_2y,\ M_2=I-X_2(X_2^{\top}X_2)^{-1}X_2^{\top}posterior = Gamma(α + Σxᵢ, β + n) (rate parameterisation)
Posterior mean (α + s)/(β + n).
Worked example: al = 2, be = 1, s = 14, n = 5 → [16, 6, 2.66667] ✓ checked
See also: Poisson distribution: mean · Beta–binomial conjugate update
\lambda\sim\mathrm{Gamma}(\alpha,\beta),\ \sum x_i=s\text{ over }n\ \Rightarrow\ \lambda\mid x\sim\mathrm{Gamma}(\alpha+s,\ \beta+n)β̂ = (XᵀΣ⁻¹X)⁻¹ XᵀΣ⁻¹y
where Σ = error covariance
Worked example: X = [[1, 0], [1, 1], [1, 2]], y = [1, 3, 5], S = [9 values] → [1, 2] ✓ checked
See also: weighted least squares
\hat\beta=(X^{\top}\Sigma^{-1}X)^{-1}X^{\top}\Sigma^{-1}\mathbf y,\quad \operatorname{Var}(\hat\beta)=(X^{\top}\Sigma^{-1}X)^{-1}x̄_g = (Π xᵢ)^(1/n)
where xᵢ > 0
The right average for growth rates and ratios.
Worked example: x = [1, 2, 4, 8] → 2.82843 ✓ checked
See also: harmonic mean · AM–GM inequality
\bar x_g=\Big(\prod_i x_i\Big)^{1/n}=\exp\!\Big(\frac1n\sum_i\ln x_i\Big)identity μ; log ln μ; logit ln(μ/(1−μ)); probit Φ⁻¹(μ)
Normal ↔ identity, Poisson ↔ log, Bernoulli ↔ logit or probit.
Worked example: eta = 0 → [0.5, 1] ✓ checked
See also: log-odds (logit) · Poisson regression log-likelihood
g(\mu)=\eta=\mathbf x^{\top}\boldsymbol\beta:\quad \text{identity }\mu,\ \ \text{log }\ln\mu,\ \ \text{logit }\ln\frac\mu{1-\mu},\ \ \text{probit }\Phi^{-1}(\mu)x̄_h = n / Σ (1/xᵢ)
where xᵢ > 0
The right average for rates (speed over equal distances) and the basis of the F₁ score.
Worked example: x = [1, 2, 4] → 1.71429 ✓ checked
See also: geometric distribution: mean · F1 score
\bar x_h=\frac{n}{\sum_i 1/x_i}H = X(XᵀX)⁻¹Xᵀ; ŷ = Hy; Σ hᵢᵢ = p
where p = number of columns of X
hᵢᵢ is the leverage of observation i.
Worked example: X = [[1, 1], [1, 2], [1, 3], [1, 4]] → 2 ✓ checked
See also: Cook's distance · leave-one-out residual shortcut
H=X(X^{\top}X)^{-1}X^{\top},\qquad \hat{\mathbf y}=H\mathbf y,\quad h_{ii}\in[0,1],\ \ \sum_ih_{ii}=pS = 1 − F; h = f/S; H = −ln S
The hazard is the instantaneous failure rate among survivors.
Worked example: lam = 2, t = 0.5 → [2, 1] ✓ checked
See also: Weibull distribution: cumulative distribution function · Kaplan–Meier estimator
S(t)=1-F(t),\qquad h(t)=\frac{f(t)}{S(t)},\qquad H(t)=\int_0^t h=-\ln S(t)g = d·(1 − 3/(4(n₁+n₂) − 9))
where d = Cohen's d
Removes the small-sample bias of d.
Worked example: d = 0.5, n1 = 10, n2 = 10 → 0.478873 ✓ checked
See also: Cohen's d (effect size)
g=d\Big(1-\frac{3}{4(n_1+n_2)-9}\Big)reject H₍ᵢ₎ while p₍ᵢ₎ ≤ α/(m − i + 1), in order of increasing p
Controls the family-wise error rate; uniformly more powerful than Bonferroni.
Worked example: p = [0.001, 0.012, 0.03, 0.2], alpha = 0.05 → 2 ✓ checked
See also: Bonferroni correction · Benjamini–Hochberg (false discovery rate)
\text{reject } H_{(i)}\text{ while } p_{(i)}\le\frac{\alpha}{m-i+1}level and trend recursions; forecast ℓₜ + h·bₜ
Worked example: y = 12, l = 10, b = 1, a = 0.5, be = 0.5 → [11.5, 1.25, 14] ✓ checked
See also: simple exponential smoothing
\ell_t=\alpha y_t+(1-\alpha)(\ell_{t-1}+b_{t-1}),\ \ b_t=\beta(\ell_t-\ell_{t-1})+(1-\beta)b_{t-1},\ \ \hat y_{t+h}=\ell_t+hb_tT̂ = Σ yᵢ/πᵢ
where πᵢ = inclusion probability
Unbiased for the population total under any design.
Worked example: y = [10, 20], pi = [0.1, 0.2] → 200 ✓ checked
See also: importance sampling
\hat T=\sum_{i\in s}\frac{y_i}{\pi_i}β̂_IV = Cov(Y, Z)/Cov(X, Z)
where Z affects Y only through X
The Wald estimator; two-stage least squares generalises it.
Worked example: cyz = 0.6, cxz = 0.3 → 2 ✓ checked
See also: simple linear regression · intention-to-treat and LATE
\hat\beta_{IV}=\frac{\operatorname{Cov}(Y,Z)}{\operatorname{Cov}(X,Z)}LATE = ITT / (share of compliers)
Worked example: itt = 2, comp = 0.5 → 4 ✓ checked
See also: instrumental-variable estimator
\mathrm{LATE}=\frac{\mathrm{ITT}}{\text{compliance rate}}=\frac{\mathbb E[Y\mid Z=1]-\mathbb E[Y\mid Z=0]}{\mathbb E[T\mid Z=1]-\mathbb E[T\mid Z=0]}IQR = Q₃ − Q₁; fences Q₁ − 1.5 IQR, Q₃ + 1.5 IQR
Worked example: x = [9 values] → 4 ✓ checked
See also: median absolute deviation · percentile (linear interpolation)
\mathrm{IQR}=Q_3-Q_1,\qquad \text{outlier fences: } Q_1-1.5\,\mathrm{IQR},\ Q_3+1.5\,\mathrm{IQR}ICC(1) = (MS_B − MS_W)/(MS_B + (k−1)MS_W)
where k = ratings per subject
Worked example: msb = 10, msw = 2, k = 3 → 0.571429 ✓ checked
See also: Cohen's kappa · design effect (cluster sampling)
\mathrm{ICC}(1)=\frac{\mathrm{MS}_B-\mathrm{MS}_W}{\mathrm{MS}_B+(k-1)\mathrm{MS}_W}τ̂ = (1/n) Σ (TᵢYᵢ/e(Xᵢ) − (1 − Tᵢ)Yᵢ/(1 − e(Xᵢ)))
where e(x) = propensity score P(T = 1 | x)
Reweights units to mimic a randomised experiment.
Worked example: T = [1, 0], Y = [10, 6], e = [0.5, 0.5] → 4 ✓ checked
See also: average treatment effect · Horvitz–Thompson estimator
\hat\tau=\frac1n\sum_i\Big(\frac{T_iY_i}{e(X_i)}-\frac{(1-T_i)Y_i}{1-e(X_i)}\Big)Var_jack = ((n−1)/n) Σ (θ̂₍₋ᵢ₎ − mean)²
where θ̂₍₋ᵢ₎ = statistic leaving out observation i
Worked example: x = [1, 2, 4, 7] → 1.75 ✓ checked
See also: bootstrap standard error
\widehat{\operatorname{Var}}_{\text{jack}}=\frac{n-1}{n}\sum_{i=1}^{n}\big(\hat\theta_{(-i)}-\bar\theta_{(\cdot)}\big)^2θ̂ = (1 − (p − 2)σ²/‖x‖²) x
where p ≥ 3 means
Shrinking toward 0 beats the raw estimate in total squared error when p ≥ 3: the origin of shrinkage and regularisation.
Worked example: x = [3, 4, 0], s2 = 1 → [2.88, 3.84, 0] ✓ checked
See also: ridge regression (closed form)
\hat\theta_{JS}=\Big(1-\frac{(p-2)\sigma^2}{\|\mathbf x\|^2}\Big)\mathbf xπ(θ) ∝ √I(θ)
where I = Fisher information
Invariant under reparameterisation. For a Bernoulli, Beta(½, ½).
Worked example: p = 0.25 → 2.3094 ✓ checked
See also: Fisher information (Bernoulli)
\pi(\theta)\propto\sqrt{I(\theta)}Ŝ(t) = Π (1 − dᵢ/nᵢ)
where dᵢ = events, nᵢ = at risk at time tᵢ
Handles censored observations.
Worked example: d = [1, 1, 2], n = [10, 8, 5] → 0.4725 ✓ checked
See also: hazard and survival functions
\hat S(t)=\prod_{t_i\le t}\Big(1-\frac{d_i}{n_i}\Big)τ = (C − D)/C(n, 2)
where C, D = concordant and discordant pairs
Worked example: C = 8, D = 2 → 0.6 ✓ checked
See also: Spearman's rank correlation
\tau=\frac{C-D}{\binom n2}D_n = sup_x |Fₙ(x) − F(x)|
where Fₙ = empirical cdf
Distribution-free under the null; reject for large √n·Dₙ.
Worked example: x = [0.1, 0.4, 0.7] → 0.3 ✓ checked
See also: empirical distribution function
D_n=\sup_x\big|F_n(x)-F(x)\big|H = 12/(N(N+1)) Σ R_g²/n_g − 3(N + 1)
where R_g = rank sum of group g
Nonparametric one-way ANOVA.
Worked example: R = [15, 25, 38], n = [4, 4, 4] → 5.11538 ✓ checked
See also: one-way ANOVA F statistic
H=\frac{12}{N(N+1)}\sum_g\frac{R_g^2}{n_g}-3(N+1)γ₂ = E[((X − μ)/σ)⁴] − 3
Zero for the normal. Positive means heavier tails than the normal.
Worked example: x = [1, 2, 3, 4, 10] → -0.212 ✓ checked
See also: skewness
\gamma_2=\mathbb E\Big[\Big(\frac{X-\mu}{\sigma}\Big)^4\Big]-3β̂ⱼ = sign(β̂_OLS)(|β̂_OLS| − λ)₊
where (·)₊ = max(·, 0)
The soft-thresholding operator: small coefficients become exactly zero.
Worked example: b = [3, -0.5, 1.2], lam = 1 → [2, -0, 0.2] ✓ checked
See also: L1 penalty (lasso) · proximal operator of the L1 norm
\hat\beta_j=\operatorname{sign}(\hat\beta_j^{\mathrm{OLS}})\big(|\hat\beta_j^{\mathrm{OLS}}|-\lambda\big)_+P(|X̄ₙ − μ| ≥ ε) ≤ σ²/(nε²) → 0
Weak law via Chebyshev. The strong law says X̄ₙ → μ with probability 1.
Worked example: s2 = 4, n = 100, e = 0.5 → 0.16 ✓ checked
See also: Chebyshev's inequality · central limit theorem (standardised mean)
P\big(|\bar X_n-\mu|\ge\varepsilon\big)\le\frac{\sigma^2}{n\varepsilon^2}\ \xrightarrow{n\to\infty}\ 0standard error ∝ 1/√n
Worked example: n1 = 100, n2 = 400 → 0.5 ✓ checked
See also: Monte Carlo standard error · standard error of the mean
\text{error}\propto\frac1{\sqrt n}\ (\text{noise-limited}),\qquad \text{quadrupling the data halves the standard error}e₍₋ᵢ₎ = eᵢ/(1 − hᵢᵢ)
Exact leave-one-out error for linear smoothers without refitting.
Worked example: e = 0.5, h = 0.25 → 0.666667 ✓ checked
See also: hat matrix and leverage
e_{(-i)}=\frac{e_i}{1-h_{ii}},\qquad \mathrm{PRESS}=\sum_i\Big(\frac{e_i}{1-h_{ii}}\Big)^2L(θ) = Π f(xᵢ; θ); ℓ = Σ ln f
Bernoulli: p̂ = (number of successes)/n.
Worked example: k = 7, n = 10 → 0.7 ✓ checked
See also: maximum likelihood estimates of a Gaussian · Fisher information
L(\theta)=\prod_if(x_i;\theta),\quad \ell(\theta)=\sum_i\ln f(x_i;\theta),\quad \hat\theta=\arg\max_\theta\ell,\ \ \ell'(\hat\theta)=0−2 ln Λ = 2(ℓ(θ̂₁) − ℓ(θ̂₀)) → χ²(q)
where q = difference in free parameters (Wilks' theorem)
Worked example: l1 = -100, l0 = -104 → 8 ✓ checked
See also: Akaike information criterion
-2\ln\Lambda=2\big(\ell(\hat\theta_1)-\ell(\hat\theta_0)\big)\ \xrightarrow{d}\ \chi^2_{q}Q = n(n + 2) Σ ρ̂ₖ²/(n − k)
Tests whether residual autocorrelations are jointly zero.
Worked example: n = 100, rho = [0.2, 0.1] → 5.16203 ✓ checked
See also: autocorrelation function
Q=n(n+2)\sum_{k=1}^{h}\frac{\hat\rho_k^2}{n-k}\ \sim\ \chi^2_hlogit(p) = ln( p / (1 − p) )
The inverse of the sigmoid. Logistic regression is linear in the log-odds.
Worked example: p = 0.8 → 1.38629 ✓ checked
See also: sigmoid (logistic function) · logistic regression model
\operatorname{logit}(p)=\ln\frac{p}{1-p},\qquad \sigma(\operatorname{logit}(p))=pχ² = (O₁ − E₁)²/V
Compares survival curves of two groups.
Worked example: o = 12, e = 8, v = 4 → 4 ✓ checked
See also: Kaplan–Meier estimator
\chi^2=\frac{(O_1-E_1)^2}{V},\quad E_1=\sum_t\frac{n_{1t}d_t}{n_t}ℓ(w) = Σ [yᵢ log pᵢ + (1 − yᵢ) log(1 − pᵢ)]
Concave in w; maximised by Newton / IRLS.
Worked example: p = [0.8, 0.3], y = [1, 0] → -0.579818 ✓ checked
See also: Newton / IRLS step for logistic regression · binary cross-entropy (log loss)
\ell(\mathbf w)=\sum_i\big[y_i\log p_i+(1-y_i)\log(1-p_i)\big],\quad p_i=\sigma(\mathbf w^{\top}\mathbf x_i)xₜ = εₜ + θεₜ₋₁; ρ₁ = θ/(1 + θ²)
Worked example: th = 0.5 → 0.4 ✓ checked
See also: AR(1) process
x_t=\varepsilon_t+\theta\varepsilon_{t-1},\quad \rho_1=\frac{\theta}{1+\theta^2},\ \rho_k=0\ (k\ge2)Cp = SSE_p/σ̂² − n + 2p
A good subset model has Cp close to p.
Worked example: sse = 50, s2 = 2, n = 30, p = 4 → 3 ✓ checked
See also: Akaike information criterion
C_p=\frac{\mathrm{SSE}_p}{\hat\sigma^2}-n+2pU₁ = R₁ − n₁(n₁+1)/2; z = (U − n₁n₂/2)/√(n₁n₂(n₁+n₂+1)/12)
where R₁ = rank sum of group 1
Distribution-free alternative to the two-sample t test.
Worked example: r1 = 40, n1 = 7, n2 = 8 → [12, -1.85164] ✓ checked
See also: Welch's t test
U_1=R_1-\frac{n_1(n_1+1)}{2},\quad z=\frac{U-n_1n_2/2}{\sqrt{n_1n_2(n_1+n_2+1)/12}}μ̂_MAP = ( (n/σ²) x̄ + μ₀/τ² ) / ( n/σ² + 1/τ² )
where likelihood N(μ, σ²), prior N(μ₀, τ²), n samples with mean x̄
A precision-weighted average of the data mean and the prior mean; the prior fades as n grows.
Worked example: n = 10, s2 = 4, xbar = 2, mu0 = 0, t2 = 1 → 1.42857 ✓ checked
See also: Bayes' theorem · maximum likelihood estimates of a Gaussian · L2 penalty (weight decay)
\hat\mu_{\mathrm{MAP}}=\frac{\frac{n}{\sigma^2}\bar x+\frac{1}{\tau^2}\mu_0}{\frac{n}{\sigma^2}+\frac{1}{\tau^2}}μ̂ = (1/n) Σ xᵢ σ̂² = (1/n) Σ (xᵢ − μ̂)²
σ̂² is biased (divides by n); see sample_variance.
Worked example: x = [2, 4, 4, 4, 5, 5, 7, 9] → 4 ✓ checked
See also: sample variance (Bessel's correction)
\hat\mu=\frac1n\sum x_i,\qquad \hat\sigma^2=\frac1n\sum (x_i-\hat\mu)^2P(f − Ef ≥ t) ≤ exp(−2t²/Σcᵢ²)
where changing coordinate i alters f by at most cᵢ
Bounded differences; underlies many generalisation bounds.
Worked example: t = 5, c = [25 values] → 0.135335 ✓ checked
See also: Hoeffding's inequality
P\big(f-\mathbb Ef\ge t\big)\le\exp\!\Big(-\frac{2t^2}{\sum_ic_i^2}\Big)χ² = (b − c)²/(b + c)
where b, c = discordant pairs
Compares two classifiers (or paired binary outcomes) on the same test set.
Worked example: b = 15, c = 5 → 5 ✓ checked
See also: paired t test
\chi^2=\frac{(b-c)^2}{b+c}\quad(\nu=1)median = middle order statistic (average of the middle two if n is even)
where x₍ᵢ₎ = i-th smallest
Minimises Σ|xᵢ − c|; robust to outliers.
Worked example: x = [7, 1, 3, 9, 5, 100] → 6 ✓ checked
See also: sample mean · median absolute deviation
\tilde x=\begin{cases}x_{((n+1)/2)}&n\text{ odd}\\ \tfrac12\big(x_{(n/2)}+x_{(n/2+1)}\big)&n\text{ even}\end{cases}MAD = median(|xᵢ − median|); σ̂ ≈ 1.4826·MAD
A robust spread; the 1.4826 factor makes it consistent with σ for normal data.
Worked example: x = [1, 2, 3, 4, 100] → 1 ✓ checked
See also: median · interquartile range
\mathrm{MAD}=\operatorname{median}\big(|x_i-\tilde x|\big),\qquad \hat\sigma\approx1.4826\,\mathrm{MAD}indirect = a·b; total = direct + indirect
where a: X → M, b: M → Y given X
Worked example: a = 0.5, b = 0.8, cp = 0.3 → [0.4, 0.7] ✓ checked
See also: average treatment effect
\text{indirect effect}=a\,b,\qquad \text{total}=c'+abMDE = (z_{α/2} + z_β) σ √(2/n)where n per arm
Worked example: za = 1.96, zb = 0.8416, s = 10, n = 1000 → 1.25291 ✓ checked
See also: sample size for comparing two means · power of a one-sided z test
\mathrm{MDE}=(z_{1-\alpha/2}+z_{1-\beta})\,\sigma\sqrt{\frac2n}w ← w + (XᵀRX)⁻¹ Xᵀ(y − p)
Converges in a handful of iterations.
Worked example: X = [[1, 0], [1, 1]], y = [0, 1], w = [0, 0] → [-2, 4] ✓ checked
See also: logistic regression log-likelihood · Newton's method (multivariate)
\mathbf w\leftarrow\mathbf w+(X^{\top}RX)^{-1}X^{\top}(\mathbf y-\mathbf p),\quad R=\operatorname{diag}\big(p_i(1-p_i)\big)n_h = n · N_h S_h / Σ N_k S_k
Optimal stratified sample sizes for a fixed total n.
Worked example: N = [100, 200], Sd = [2, 1], n = 60 → [30, 30] ✓ checked
See also: variance of a sample mean (simple random sampling)
n_h=n\,\frac{N_h S_h}{\sum_k N_kS_k}z_p = Φ⁻¹(p); 68–95–99.7 rule
z₀.₉₇₅ = 1.96, z₀.₉₅ = 1.645, z₀.₉₉₅ = 2.576.
Worked example: p = 0.975 → [1.95996, 0.682689, 0.9545] ✓ checked
See also: normal cumulative distribution · z-score
z_p=\Phi^{-1}(p),\qquad P(|Z|<k)=\operatorname{erf}\!\Big(\frac k{\sqrt2}\Big)\ \ (\approx68\%,\ 95\%,\ 99.7\%\text{ for }k=1,2,3)precision-weighted mean; posterior precision = prior precision + n/σ²
where prior N(μ₀, τ²), data variance σ²
Worked example: m0 = 0, t2 = 1, xbar = 2, s2 = 4, n = 10 → [1.42857, 0.285714] ✓ checked
See also: maximum a posteriori (Gaussian mean, Gaussian prior)
\mu\mid x\sim\mathcal N\Big(\frac{\mu_0/\tau^2+n\bar x/\sigma^2}{1/\tau^2+n/\sigma^2},\ \Big(\frac1{\tau^2}+\frac n{\sigma^2}\Big)^{-1}\Big)odds = p/(1 − p); p = odds/(1 + odds)
Worked example: p = 0.8 → [4, 0.8] ✓ checked
See also: log-odds (logit) · odds ratio and relative risk
\text{odds}=\frac p{1-p},\qquad p=\frac{\text{odds}}{1+\text{odds}},\qquad \text{log-odds}=\ln\frac p{1-p}OR = ad/(bc); RR = (a/(a+b)) / (c/(c+d))
where table [[a, b], [c, d]]: exposed (events a, none b), unexposed (events c, none d)
OR approximates RR only when events are rare.
Worked example: a = 20, b = 80, c = 10, d = 90 → [2.25, 2] ✓ checked
See also: log-odds (logit)
\mathrm{OR}=\frac{ad}{bc},\qquad \mathrm{RR}=\frac{a/(a+b)}{c/(c+d)}t = (x̄ − μ₀) / (s/√n)
where μ₀ = hypothesised mean; n − 1 degrees of freedom
Worked example: xbar = 10.5, mu0 = 10, s = 2, n = 100 → 2.5 ✓ checked
See also: z-score · standard error of the mean
t=\frac{\bar x-\mu_0}{s/\sqrt n}z = (x̄ − μ₀)/(σ/√n)
where σ known
Worked example: xbar = 52, mu0 = 50, s = 10, n = 100 → 2 ✓ checked
See also: one-sample t statistic · two-sided p-value from a z statistic
z=\frac{\bar x-\mu_0}{\sigma/\sqrt n}F = (SS_B/(k−1)) / (SS_W/(N−k))
where k groups, N observations
Worked example: g1 = [1, 2, 3], g2 = [2, 3, 4], g3 = [5, 6, 7] → 13 ✓ checked
See also: eta squared (effect size) · F distribution distribution: mean
F=\frac{\mathrm{SS}_B/(k-1)}{\mathrm{SS}_W/(N-k)},\quad \mathrm{SS}_B=\sum_g n_g(\bar x_g-\bar x)^2,\ \mathrm{SS}_W=\sum_g\sum_i(x_{gi}-\bar x_g)^2F = (R²/p) / ((1 − R²)/(n − p − 1))
where p predictors
Worked example: r2 = 0.5, p = 3, n = 54 → 16.6667 ✓ checked
See also: coefficient of determination R² · one-way ANOVA F statistic
F=\frac{R^2/p}{(1-R^2)/(n-p-1)}t = d̄ / (s_d/√n)
where dᵢ = paired differences
A one-sample t test on the differences.
Worked example: d = [1, 2, 0, 3, 4] → 2.82843 ✓ checked
See also: one-sample t statistic
t=\frac{\bar d}{s_d/\sqrt n},\quad d_i=x_i-y_iP(X > θEX) ≥ (1 − θ)² (EX)²/E[X²]
where X ≥ 0, 0 ≤ θ ≤ 1
A lower bound on how often X is not small.
Worked example: th = 0.5, m1 = 2, m2 = 5 → 0.2 ✓ checked
See also: Markov's inequality
P\big(X>\theta\,\mathbb E X\big)\ge(1-\theta)^2\frac{(\mathbb E X)^2}{\mathbb E[X^2]}ρ = Cov(X,Y) / (σₓ σᵧ)
where ρ ∈ [−1, 1]
Measures linear association only. A perfect nonlinear relationship can give ρ = 0.
Worked example: x = [1, 2, 3, 4], y = [2, 4, 5, 9] → 0.964764 ✓ checked
See also: covariance · coefficient of determination R²
\rho_{XY} = \frac{\operatorname{Cov}(X,Y)}{\sigma_X\,\sigma_Y}h = 1 + (n−1)p/100; interpolate between the neighbouring order statistics
The default of NumPy and R type 7.
Worked example: x = [10, 20, 30, 40], p = 30 → 19 ✓ checked
See also: interquartile range
x_{(p)}=x_{(\lfloor h\rfloor)}+(h-\lfloor h\rfloor)\big(x_{(\lfloor h\rfloor+1)}-x_{(\lfloor h\rfloor)}\big),\quad h=1+(n-1)\tfrac p{100}p = (1 + #{T*_b ≥ T_obs}) / (1 + B)The +1 counts the observed data as one permutation, so p is never 0.
Worked example: k = 7, B = 999 → 0.008 ✓ checked
See also: bootstrap standard error
p=\frac{1+\#\{b:\ T^{*}_b\ge T_{\text{obs}}\}}{1+B}φ = (ad − bc)/√((a+b)(c+d)(a+c)(b+d))
where 2×2 table [[a, b], [c, d]]
The same as the Matthews correlation coefficient.
Worked example: a = 20, b = 30, c = 30, d = 20 → -0.2 ✓ checked
See also: Matthews correlation coefficient · Cramér's V
\phi=\frac{ad-bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}ℓ = Σ [yᵢ log μᵢ − μᵢ − log yᵢ!]
Log link keeps the mean positive.
Worked example: mu = [2, 3], y = [1, 4] → -3.09046 ✓ checked
See also: Poisson distribution · GLM link functions
\ell=\sum_i\big[y_i\log\mu_i-\mu_i-\log y_i!\big],\quad \mu_i=e^{\mathbf w^{\top}\mathbf x_i}s_p² = ((n₁−1)s₁² + (n₂−1)s₂²) / (n₁ + n₂ − 2)
Assumes equal population variances.
Worked example: n1 = 10, n2 = 15, s1 = 4, s2 = 9 → 7.04348 ✓ checked
See also: two-sample t test (pooled) · Welch's t test
s_p^2=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}power = Φ(δ√n/σ − z_{1−α})where δ = true effect, σ = standard deviation
Worked example: d = 0.5, n = 50, s = 1, za = 1.6449 → 0.970663 ✓ checked
See also: type I and II errors, and power · sample size for comparing two means
\text{power}=\Phi\!\Big(\frac{\delta\sqrt n}{\sigma}-z_{1-\alpha}\Big)ρ_τ(u) = u(τ − 1[u < 0]), u = y − ŷ
Minimised by the τ-quantile; the loss of quantile regression.
Worked example: y = 3, yh = 5, tau = 0.9 → 0.2 ✓ checked
See also: mean absolute error
\rho_\tau(u)=u\big(\tau-\mathbf 1[u<0]\big)xₜ = xₜ₋₁ + εₜ; Var(xₜ) = tσ²
Non-stationary: variance grows linearly; difference it to get white noise.
Worked example: t = 100, s2 = 2 → 200 ✓ checked
See also: AR(1) process
x_t=x_{t-1}+\varepsilon_t\ \Rightarrow\ \operatorname{Var}(x_t)=t\,\sigma^2μ̂_y = (ȳ/x̄) μ_x
where μ_x = known population mean of x
Beats ȳ when x and y are strongly correlated.
Worked example: ybar = 50, xbar = 10, mux = 12 → 60 ✓ checked
See also: variance of a sample mean (simple random sampling)
\hat\mu_y=\frac{\bar y}{\bar x}\,\mu_xARR = p_control − p_treated; NNT = 1/ARR
Worked example: pc = 0.2, pt = 0.15 → 20 ✓ checked
See also: odds ratio and relative risk
\mathrm{ARR}=p_c-p_t,\qquad \mathrm{NNT}=\frac1{\mathrm{ARR}}upper 95% bound on a rate after zero events ≈ 3/n
Worked example: n = 300 → 0.01 ✓ checked
See also: Wilson score interval
\text{0 events in }n\text{ trials}\ \Rightarrow\ p<\frac3n\ \text{ with 95\% confidence}δ = xₙ − x̄ₙ₋₁; x̄ₙ = x̄ₙ₋₁ + δ/n; Mₙ = Mₙ₋₁ + δ(xₙ − x̄ₙ)
One pass, numerically stable; works for streaming data.
Worked example: x = [2, 4, 4, 4, 5, 5, 7, 9] → [5, 4.57143] ✓ checked
See also: sample variance (Bessel's correction)
\delta=x_n-\bar x_{n-1},\ \ \bar x_n=\bar x_{n-1}+\frac{\delta}{n},\ \ M_n=M_{n-1}+\delta\,(x_n-\bar x_n),\ \ s^2=\frac{M_n}{n-1}S = (1/(n−1)) Σ (xᵢ − x̄)(xᵢ − x̄)ᵀ
Positive semi-definite; rank at most n − 1.
Worked example: X = [[1, 2], [2, 3], [3, 7]] → [[1, 2.5], [2.5, 7]] ✓ checked
See also: covariance · principal component analysis
S=\frac1{n-1}\sum_{i=1}^{n}(\mathbf x_i-\bar{\mathbf x})(\mathbf x_i-\bar{\mathbf x})^{\top}x̄ = (1/n) Σ xᵢ
The least-squares estimate of a location: it minimises Σ(xᵢ − c)².
Worked example: x = [2, 4, 4, 4, 5, 5, 7, 9] → 5 ✓ checked
See also: weighted mean · median
\bar x=\frac1n\sum_{i=1}^{n}x_in = 2 (z_{1−α/2} + z_{1−β})² σ² / δ² per groupwhere δ = smallest difference worth detecting, σ = standard deviation
z = 1.96 for α = 0.05 (two-sided), 0.8416 for 80% power.
Worked example: za = 1.96, zb = 0.8416, s = 1, d = 0.5 → 62.7917 ✓ checked
See also: type I and II errors, and power · Cohen's d (effect size)
n=\frac{2\,(z_{1-\alpha/2}+z_{1-\beta})^2\,\sigma^2}{\delta^2}\ \text{per group}n = z² p(1−p)/E²
where E = margin of error; use p = 0.5 if unknown
Worked example: z = 1.96, p = 0.5, E = 0.03 → 1067.11 ✓ checked
See also: finite population correction
n=\frac{z^2\,p(1-p)}{E^2}s² = (1/(n−1)) Σ (xᵢ − x̄)²
where n − 1 makes it an unbiased estimator of σ²
Worked example: x = [2, 4, 4, 4, 5, 5, 7, 9] → 4.57143 ✓ checked
See also: variance and standard deviation · maximum likelihood estimates of a Gaussian
s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar x)^2χ² test on the arm sizes
A tiny p-value means the assignment is broken and results are untrustworthy.
Worked example: O = [5100, 4900], E = [5000, 5000] → 4 ✓ checked
See also: chi-square goodness of fit
\chi^2=\sum_i\frac{(O_i-E_i)^2}{E_i}\ \ \text{against the intended allocation}stop as soon as the log-likelihood ratio leaves (ln(β/(1−α)), ln((1−β)/α))
Needs on average fewer samples than any fixed-sample test of the same error rates (Wald–Wolfowitz).
Worked example: a = 0.05, b = 0.2 → [2.77259, -1.55814] ✓ checked
See also: likelihood ratio test
\Lambda_n=\sum_i\ln\frac{f_1(x_i)}{f_0(x_i)};\quad \text{accept }H_1\text{ if }\Lambda_n\ge\ln\frac{1-\beta}\alpha,\quad \text{accept }H_0\text{ if }\Lambda_n\le\ln\frac\beta{1-\alpha}ŷₜ₊₁ = αyₜ + (1 − α)ŷₜ
where 0 < α < 1
Equivalent to an exponentially weighted moving average.
Worked example: y = 12, yh = 10, a = 0.3 → 10.6 ✓ checked
See also: Holt's linear trend method · EWMA volatility (RiskMetrics)
\hat y_{t+1}=\alpha y_t+(1-\alpha)\hat y_tβ̂₁ = Σ(xᵢ−x̄)(yᵢ−ȳ)/Σ(xᵢ−x̄)² = r·s_y/s_x; β̂₀ = ȳ − β̂₁x̄
Worked example: x = [1, 2, 3, 4], y = [2, 4, 5, 9] → [2.2, -0.5] ✓ checked
See also: ordinary least squares (normal equations) · Pearson correlation
\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}=r\frac{s_y}{s_x},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar xγ₁ = E[((X − μ)/σ)³]
Zero for symmetric distributions; positive means a long right tail.
Worked example: x = [1, 2, 3, 4, 10] → 1.13842 ✓ checked
See also: kurtosis (excess)
\gamma_1=\mathbb E\Big[\Big(\frac{X-\mu}{\sigma}\Big)^3\Big]Xₙ → X in distribution, Yₙ → c in probability ⇒ Xₙ + Yₙ → X + c, XₙYₙ → cX
Why replacing σ by s in a z statistic is harmless for large n.
Worked example: c = 2, x = 1.5 → 3 ✓ checked
See also: central limit theorem (standardised mean)
X_n\xrightarrow{d}X,\ Y_n\xrightarrow{p}c\ \Rightarrow\ X_n+Y_n\xrightarrow{d}X+c,\ \ X_nY_n\xrightarrow{d}cXρ_s = 1 − 6Σdᵢ²/(n(n² − 1))
where dᵢ = rank difference (no ties)
Pearson correlation of the ranks; detects monotone, not just linear, relationships.
Worked example: d2 = 4, n = 5 → 0.8 ✓ checked
See also: Pearson correlation · Kendall's tau
\rho_s=1-\frac{6\sum_i d_i^2}{n(n^2-1)}SE = √(p̂(1 − p̂)/n)
Worked example: p = 0.3, n = 100 → 0.0458258 ✓ checked
See also: Wald interval for a proportion
\mathrm{SE}(\hat p)=\sqrt{\frac{\hat p(1-\hat p)}{n}}SE = σ / √n
Replace σ with the sample standard deviation s when σ is unknown.
Worked example: s = 2, n = 100 → 0.2 ✓ checked
See also: confidence interval for a mean · central limit theorem (standardised mean)
\mathrm{SE}(\bar X)=\frac{\sigma}{\sqrt n}SE(β̂₁) = s/√Σ(xᵢ − x̄)²
Worked example: x = [1, 2, 3, 4], y = [2, 4, 5, 9] → 0.424264 ✓ checked
See also: simple linear regression
\mathrm{SE}(\hat\beta_1)=\frac{s}{\sqrt{\sum(x_i-\bar x)^2}},\quad s^2=\frac{\sum(y_i-\hat y_i)^2}{n-2}Z = Σ wᵢzᵢ / √(Σ wᵢ²)
where zᵢ = Φ⁻¹(1 − pᵢ); weights often √nᵢ
Worked example: z = [1.5, 2], w = [1, 1] → 2.47487 ✓ checked
See also: Fisher's method for combining p-values
Z=\frac{\sum_i w_iz_i}{\sqrt{\sum_i w_i^2}}f(x; θ) = g(T(x), θ) h(x)
For i.i.d. Bernoulli data the count of successes Σ xᵢ is sufficient. Rao–Blackwell: conditioning an estimator on a sufficient statistic never increases its variance.
Worked example: x = [1, 0, 1, 1], p = 0.6 → 0.0864 ✓ checked
See also: likelihood and log-likelihood
f(x;\theta)=g\big(T(x),\theta\big)\,h(x)\ \iff\ T\text{ is sufficient for }\thetax̄ ± t·s/√n
Use the t quantile (2.262 for n = 10 at 95%), not 1.96, for small samples.
Worked example: xbar = 10, s = 2, n = 10, t = 2.262 → [8.56939, 11.4306] ✓ checked
See also: confidence interval for a mean · Student's t distribution: density
\bar x\pm t_{n-1,\,1-\alpha/2}\frac{s}{\sqrt n}q = (x̄ᵢ − x̄ⱼ)/√(MS_W/n)
where n = per-group size
Post-hoc pairwise comparisons after an ANOVA, controlling the family-wise error.
Worked example: d = 3, msw = 4, n = 9 → 4.5 ✓ checked
See also: one-way ANOVA F statistic · Holm–Bonferroni procedure
q=\frac{\bar x_i-\bar x_j}{\sqrt{\mathrm{MS}_W/n}}z = (p̂₁ − p̂₂)/√(p̂(1−p̂)(1/n₁ + 1/n₂))
A/B tests on conversion rates.
Worked example: x1 = 60, n1 = 200, x2 = 40, n2 = 200 → 2.3094 ✓ checked
See also: standard error of a proportion
z=\frac{\hat p_1-\hat p_2}{\sqrt{\hat p(1-\hat p)\big(\frac1{n_1}+\frac1{n_2}\big)}},\quad \hat p=\frac{x_1+x_2}{n_1+n_2}t = (x̄₁ − x̄₂)/(s_p√(1/n₁ + 1/n₂))
Assumes equal variances.
Worked example: m1 = 10, m2 = 8, sp2 = 7, n1 = 20, n2 = 30 → 2.61861 ✓ checked
See also: pooled variance · Welch's t test
t=\frac{\bar x_1-\bar x_2}{s_p\sqrt{1/n_1+1/n_2}},\quad \nu=n_1+n_2-2p = 2 (1 − Φ(|z|))
where Φ = standard normal cdf
The probability, under the null, of a result at least this extreme. It is not the probability the null is true.
Worked example: z = 1.96 → 0.0499958 ✓ checked
See also: normal cumulative distribution · one-sample t statistic · type I and II errors, and power
p=2\big(1-\Phi(|z|)\big)α = P(reject H₀ | H₀ true); β = P(keep H₀ | H₁ true); power = 1 − β
Type I = false positive, type II = false negative. Lowering α lowers power unless n grows.
Worked example: beta = 0.2 → 0.8 ✓ checked
See also: two-sided p-value from a z statistic · sample size for comparing two means · Bonferroni correction
\alpha=P(\text{reject }H_0\mid H_0),\quad \beta=P(\text{fail to reject}\mid H_1),\quad \text{power}=1-\betaVIFⱼ = 1/(1 − Rⱼ²)
where Rⱼ² = R² from regressing predictor j on the others
Above about 10 signals harmful multicollinearity.
Worked example: r2 = 0.9 → 10 ✓ checked
See also: ordinary least squares (normal equations)
\mathrm{VIF}_j=\frac1{1-R_j^2}Var(x̄) = (1 − n/N) S²/n
where S² = population variance (N − 1 divisor)
Worked example: n = 100, N = 1000, S2 = 25 → 0.225 ✓ checked
See also: finite population correction
\operatorname{Var}(\bar x)=\Big(1-\frac nN\Big)\frac{S^2}nσ² = Σ n_g (σ_g² + (μ_g − μ)²) / Σ n_g
where μ = overall mean
Within-group plus between-group spread.
Worked example: n = [10, 20], m = [1, 4], v = [2, 3] → 4.66667 ✓ checked
See also: law of total variance
\sigma^2=\frac{\sum_g n_g(\sigma_g^2+(\mu_g-\mu)^2)}{\sum_g n_g}p̂ ± z√(p̂(1−p̂)/n)
Poor for small n or p near 0 or 1; prefer Wilson.
Worked example: p = 0.3, n = 100, z = 1.96 → [0.210182, 0.389818] ✓ checked
See also: Wilson score interval
\hat p\pm z_{1-\alpha/2}\sqrt{\frac{\hat p(1-\hat p)}{n}}W = (θ̂ − θ₀)²/Var(θ̂)
The square of a z statistic.
Worked example: th = 1.2, t0 = 1, v = 0.01 → 4 ✓ checked
See also: likelihood ratio test
W=\frac{(\hat\theta-\theta_0)^2}{\widehat{\operatorname{Var}}(\hat\theta)}\ \sim\ \chi^2_1β̂ = (XᵀWX)⁻¹ XᵀWy
where wᵢ ∝ 1/Var(εᵢ)
Optimal when errors are independent with unequal variances.
Worked example: X = [[1, 0], [1, 1], [1, 2]], y = [1, 3, 5], w = [1, 2, 1] → [1, 2] ✓ checked
See also: ordinary least squares (normal equations) · generalised least squares
\hat\beta=(X^{\top}WX)^{-1}X^{\top}W\mathbf y,\quad W=\operatorname{diag}(w_i)x̄_w = Σ wᵢxᵢ / Σ wᵢ
where wᵢ = weights
Worked example: x = [1, 2, 3], w = [3, 2, 1] → 1.66667 ✓ checked
See also: sample mean
\bar x_w=\frac{\sum_i w_ix_i}{\sum_i w_i}t = (x̄₁ − x̄₂)/√(s₁²/n₁ + s₂²/n₂); ν = Welch–Satterthwaite
Does not assume equal variances; prefer it by default.
Worked example: m1 = 10, m2 = 8, v1 = 4, v2 = 9, n1 = 20, n2 = 30 → [2.82843, 47.9965] ✓ checked
See also: two-sample t test (pooled)
t=\frac{\bar x_1-\bar x_2}{\sqrt{s_1^2/n_1+s_2^2/n_2}},\qquad \nu\approx\frac{\big(s_1^2/n_1+s_2^2/n_2\big)^2}{\frac{(s_1^2/n_1)^2}{n_1-1}+\frac{(s_2^2/n_2)^2}{n_2-1}}W = Σ ranks of positive differences; E[W] = n(n+1)/4
Nonparametric paired test.
Worked example: n = 10 → [27.5, 96.25] ✓ checked
See also: paired t test
W=\sum_{i:\,d_i>0}R_i,\qquad \mathbb E[W]=\frac{n(n+1)}{4},\ \ \operatorname{Var}(W)=\frac{n(n+1)(2n+1)}{24}Wilson: (p̂ + z²/2n ± z√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n)
Behaves well for small n and extreme proportions.
Worked example: p = 0.3, n = 100, z = 1.96 → [0.218948, 0.39585] ✓ checked
See also: Wald interval for a proportion
\frac{\hat p+\frac{z^2}{2n}\pm z\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}n}[[1, ρ₁], [ρ₁, 1]] [φ₁, φ₂] = [ρ₁, ρ₂]
Estimates AR coefficients from autocorrelations.
Worked example: r1 = 0.6, r2 = 0.4 → [0.5625, 0.0625] ✓ checked
See also: AR(1) process
\begin{pmatrix}1&\rho_1\\ \rho_1&1\end{pmatrix}\begin{pmatrix}\phi_1\\ \phi_2\end{pmatrix}=\begin{pmatrix}\rho_1\\ \rho_2\end{pmatrix}z = (x − μ) / σ
Standardises a value to units of standard deviations from the mean.
Worked example: x = 75, mu = 70, s = 10 → 0.5 ✓ checked
See also: normal cumulative distribution
z=\frac{x-\mu}{\sigma}α_each = 1 − (1 − α)^(1/m)
Slightly less conservative than Bonferroni (exact for independent tests).
Worked example: alpha = 0.05, m = 10 → 0.0051162 ✓ checked
See also: Bonferroni correction · Holm–Bonferroni procedure
\alpha_{\text{each}}=1-(1-\alpha)^{1/m}BC = Σ √(pq); D_B = −ln BC
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 0.111572 ✓ checked
See also: Hellinger distance
\mathrm{BC}=\sum_x\sqrt{p(x)q(x)},\qquad D_B=-\ln\mathrm{BC}H(p) = −p log p − (1−p) log(1−p)
Peaks at p = ½ (1 bit).
Worked example: p = 0.25 → 0.562335 ✓ checked
See also: Shannon entropy
H_b(p)=-p\log p-(1-p)\log(1-p)bits = nats / ln 2; PPL = 2^(bits) = e^(nats)
Most deep-learning libraries report nats; papers on compression report bits.
Worked example: h = 2 → [2.88539, 7.38906] ✓ checked
See also: perplexity
1\ \text{nat}=\frac1{\ln2}\ \text{bits}\approx1.4427\ \text{bits},\qquad \mathrm{PPL}=2^{H_{\text{bits}}}=e^{H_{\text{nats}}}C = 1 − ε
where ε = erasure probability
Worked example: e = 0.3 → 0.7 ✓ checked
See also: capacity of a binary symmetric channel
C=1-\varepsilonC = 1 − H_b(p)
where p = crossover probability
Worked example: p = 0.11 → 0.500084 ✓ checked
See also: binary entropy · capacity of a binary erasure channel
C=1-H_b(p)\ \text{bits per use}H(X,Y) = H(X) + H(Y|X)
Generalises to H(X₁…Xₙ) = Σ H(Xᵢ | X₁…Xᵢ₋₁): the basis of autoregressive likelihoods.
Worked example: J = [[0.4, 0.1], [0.1, 0.4]] → 1.19355 ✓ checked
See also: joint entropy · conditional entropy
H(X,Y)=H(X)+H(Y\mid X)χ²(p‖q) = Σ (p − q)²/q
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 1.77778 ✓ checked
See also: Kullback–Leibler divergence
\chi^2(p\|q)=\sum_x\frac{(p(x)-q(x))^2}{q(x)}H(Y|X) = H(X,Y) − H(X)
Remaining uncertainty about Y once X is known.
Worked example: J = [[0.4, 0.1], [0.1, 0.4]] → 0.500402 ✓ checked
See also: mutual information · Shannon entropy
H(Y\mid X)=H(X,Y)-H(X)H(p,q) = −Σₓ p(x) log q(x)
where p = true distribution, q = model distribution
H(p,q) = H(p) + KL(p‖q). With one-hot p it is the log-loss −log q(true class).
Worked example: p = [1, 0], q = [0.8, 0.2] → 0.223144 ✓ checked
See also: Kullback–Leibler divergence · Shannon entropy · softmax cross-entropy (from logits)
H(p,q)=-\sum_x p(x)\log q(x)H(p, q) = ½ ln(2πσ_q²) + (σ_p² + (μ_p − μ_q)²)/(2σ_q²)
Equals h(p) + KL(p‖q).
Worked example: mp = 0, sp = 1, mq = 1, sq = 2 → 1.86209 ✓ checked
See also: KL divergence between two Gaussians · differential entropy of a Gaussian
H(p,q)=\tfrac12\ln(2\pi\sigma_q^2)+\frac{\sigma_p^2+(\mu_p-\mu_q)^2}{2\sigma_q^2}X → Y → Z ⇒ I(X;Z) ≤ I(X;Y)
where Markov chain
Processing cannot create information. Example: two binary symmetric channels in a row.
Worked example: e1 = 0.1, e2 = 0.2 → 0.247974 ✓ checked
See also: mutual information
X\to Y\to Z\ \Rightarrow\ I(X;Z)\le I(X;Y)h = ½ ln(2πe σ²)
Differential entropy can be negative.
Worked example: s = 1 → 1.41894 ✓ checked
See also: normal (Gaussian) density · Shannon entropy
h(X)=\tfrac12\ln\!\big(2\pi e\,\sigma^2\big)h = ½ ln((2πe)ᵏ |Σ|)
where k = dimension
Worked example: S = [[1, 0], [0, 1]] → 2.83788 ✓ checked
See also: differential entropy of a Gaussian
h(\mathbf x)=\tfrac12\ln\big((2\pi e)^k|\Sigma|\big)H = −Σᵢ πᵢ Σⱼ Pᵢⱼ log Pᵢⱼ
where π = stationary distribution
Worked example: P = [[0.9, 0.1], [0.5, 0.5]], pi = [0.833333, 0.166667] → 0.386427 ✓ checked
See also: stationary distribution of a Markov chain
H=-\sum_i\pi_i\sum_jP_{ij}\log P_{ij}D_f(p‖q) = Σ q f(p/q)
f(t) = t log t gives KL, ½|t − 1| gives TV, (√t − 1)² gives (twice) squared Hellinger.
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 0.510826 ✓ checked
See also: Kullback–Leibler divergence · total variation distance
D_f(p\|q)=\sum_xq(x)\,f\!\Big(\frac{p(x)}{q(x)}\Big),\quad f\text{ convex},\ f(1)=0H(X|Y) ≤ H_b(Pₑ) + Pₑ log(|X| − 1)
where Pₑ = error probability of any estimator of X from Y
Links the error of any classifier to the information left about the label.
Worked example: pe = 0.1, m = 4 → 0.434944 ✓ checked
See also: binary entropy
H(X\mid Y)\le H_b(P_e)+P_e\log(|\mathcal X|-1)I(θ) = E[(∂ ln f/∂θ)²] = −E[∂² ln f/∂θ²]
For n i.i.d. samples it is n·I(θ).
Worked example: n = 25, s2 = 4 → 6.25 ✓ checked
See also: Fisher information (Bernoulli) · Cramér–Rao bound · natural gradient
I(\theta)=\mathbb E\Big[\Big(\frac{\partial\ln f(X;\theta)}{\partial\theta}\Big)^2\Big]=-\mathbb E\Big[\frac{\partial^2\ln f(X;\theta)}{\partial\theta^2}\Big]I(μ, σ) = diag(1/σ², 2/σ²)
Worked example: s = 2 → [0.25, 0.5] ✓ checked
See also: Fisher information
I(\mu,\sigma)=\begin{pmatrix}1/\sigma^2&0\\0&2/\sigma^2\end{pmatrix}H²(p, q) = 1 − Σ √(p q)
A symmetric divergence between 0 and 1.
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 0.105573 ✓ checked
See also: Bhattacharyya coefficient and distance · total variation distance
H^2(p,q)=1-\sum_x\sqrt{p(x)q(x)}min I(X;T) − β I(T;Y)
where T = compressed representation of X, β = trade-off
Keep what predicts Y, forget the rest of X.
Worked example: ixt = 1.5, ity = 1.2, b = 2 → -0.9 ✓ checked
See also: mutual information
\min_{p(t\mid x)}\ I(X;T)-\beta\,I(T;Y)I(x) = −log₂ p(x)
Entropy is the expected surprisal.
Worked example: p = 0.25 → 2 ✓ checked
See also: Shannon entropy
I(x)=-\log_2p(x)\ \text{bits}JS(p,q) = ½ KL(p‖m) + ½ KL(q‖m), m = (p+q)/2
Symmetric and bounded by ln 2 (nats).
Worked example: p = [1, 0], q = [0, 1] → 0.693147 ✓ checked
See also: Kullback–Leibler divergence
\mathrm{JS}(p,q)=\tfrac12 D_{\mathrm{KL}}(p\|m)+\tfrac12 D_{\mathrm{KL}}(q\|m),\quad m=\tfrac12(p+q)H(X,Y) = −Σ p(x,y) log p(x,y)
Worked example: J = [[0.4, 0.1], [0.1, 0.4]] → 1.19355 ✓ checked
See also: chain rule for entropy
H(X,Y)=-\sum_{x,y}p(x,y)\log p(x,y)KL(N₀‖N₁) = ½[tr(Σ₁⁻¹Σ₀) + Δμᵀ Σ₁⁻¹ Δμ − k + ln(|Σ₁|/|Σ₀|)]
Worked example: S0 = [[1, 0], [0, 1]], S1 = [[2, 0], [0, 2]], d = [1, 0] → 0.443147 ✓ checked
See also: KL divergence between two Gaussians
D_{\mathrm{KL}}=\tfrac12\Big[\operatorname{tr}(\Sigma_1^{-1}\Sigma_0)+(\boldsymbol\mu_1-\boldsymbol\mu_0)^{\top}\Sigma_1^{-1}(\boldsymbol\mu_1-\boldsymbol\mu_0)-k+\ln\frac{|\Sigma_1|}{|\Sigma_0|}\Big]KL = ln(σ₂/σ₁) + (σ₁² + (μ₁−μ₂)²) / (2σ₂²) − ½
Closed form used in VAEs and variational inference.
Worked example: m1 = 0, s1 = 1, m2 = 1, s2 = 2 → 0.443147 ✓ checked
See also: Kullback–Leibler divergence · VAE KL term (diagonal Gaussian)
D_{\mathrm{KL}}\big(\mathcal N(\mu_1,\sigma_1^2)\,\|\,\mathcal N(\mu_2,\sigma_2^2)\big)=\ln\frac{\sigma_2}{\sigma_1}+\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}-\frac12Σ 2^(−ℓᵢ) ≤ 1
where ℓᵢ = codeword lengths
A prefix code with these lengths exists iff it holds.
Worked example: l = [1, 2, 3, 3] → 1 ✓ checked
See also: source coding theorem
\sum_i2^{-\ell_i}\le1KL(p‖q) = Σₓ p(x) log( p(x)/q(x) )
Not symmetric and not a metric. ≥ 0, and 0 only when p = q.
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 0.510826 ✓ checked
See also: cross-entropy · Jensen–Shannon divergence · KL divergence between two Gaussians
D_{\mathrm{KL}}(p\,\|\,q)=\sum_x p(x)\log\frac{p(x)}{q(x)}MMD² = E k(x,x′) + E k(y,y′) − 2 E k(x,y)
where k = kernel, e.g. RBF
Zero iff the distributions match (for a characteristic kernel).
Worked example: x = [0, 1], y = [2, 3] → 1.16238 ✓ checked
See also: RBF (Gaussian) kernel
\mathrm{MMD}^2=\mathbb E\,k(x,x')+\mathbb E\,k(y,y')-2\,\mathbb E\,k(x,y)p(x) ∝ exp(Σ λₖ fₖ(x))
Fixed mean and variance gives the normal; fixed mean on [0, ∞) gives the exponential; nothing fixed gives the uniform.
Worked example: s = 1 → [1.41894, 0.176485] ✓ checked
See also: Shannon entropy
\max_pH(p)\ \text{s.t. }\mathbb E[f_k]=c_k\ \Rightarrow\ p(x)\propto\exp\Big(\sum_k\lambda_kf_k(x)\Big)H∞ = −log maxₓ p(x)
The probability of guessing the most likely outcome; used in cryptography.
Worked example: p = [0.7, 0.3] → 0.356675 ✓ checked
See also: Rényi entropy
H_\infty(X)=-\log\max_xp(x)MDL = L(model) + L(data | model)
where lengths in bits
A model is good if it compresses the data including its own description. BIC is an asymptotic form.
Worked example: lm = 120, ld = 880 → 1000 ✓ checked
See also: Bayesian information criterion
\mathrm{MDL}=L(\text{model})+L(\text{data}\mid\text{model})I(X;Y) = Σ p(x,y) log( p(x,y) / (p(x)p(y)) )
Zero iff X and Y are independent. I(X;Y) = H(X) − H(X|Y).
Worked example: J = [[0.4, 0.1], [0.1, 0.4]] → 0.192745 ✓ checked
See also: Shannon entropy · conditional entropy
I(X;Y)=\sum_{x,y}p(x,y)\log\frac{p(x,y)}{p(x)\,p(y)}I(X;Y) = H(X) + H(Y) − H(X,Y)
Worked example: hx = 0.693147, hy = 0.693147, hxy = 1.1935 → 0.192794 ✓ checked
See also: mutual information
I(X;Y)=H(X)+H(Y)-H(X,Y)=H(X)-H(X\mid Y)I(X;Y) = −½ ln(1 − ρ²)
where ρ = correlation
Zero at ρ = 0 and infinite at |ρ| = 1.
Worked example: rho = 0.8 → 0.510826 ✓ checked
See also: Pearson correlation · mutual information
I(X;Y)=-\tfrac12\ln\big(1-\rho^2\big)θ ← θ − η F⁻¹ ∇L
where F = Fisher information matrix
Steepest descent in the geometry of the model's distribution, invariant to reparameterisation.
Worked example: th = [0, 0], F = [[4, 0], [0, 1]], g = [2, 1], lr = 0.1 → [-0.05, -0.1] ✓ checked
See also: Fisher information
\theta\leftarrow\theta-\eta\,F^{-1}\nabla_\theta LPSNR = 10 log₁₀(MAX²/MSE)
where MAX = largest pixel value (255 for 8-bit)
Higher is better; common for image and video reconstruction.
Worked example: mx = 255, mse = 100 → 28.1308 ✓ checked
See also: signal-to-noise ratio in decibels · mean squared error
\mathrm{PSNR}=10\log_{10}\frac{\mathrm{MAX}^2}{\mathrm{MSE}}PPL = exp( −(1/N) Σ log q(xᵢ) )
where q(xᵢ) = probability the model gave the observed token
The effective number of equally likely choices per token. Lower is better.
Worked example: q = [0.5, 0.25, 0.125] → 4 ✓ checked
See also: cross-entropy
\mathrm{PPL}=\exp\!\Big(-\frac1N\sum_{i=1}^{N}\log q(x_i)\Big)TV ≤ √(KL/2) (KL in nats)
Small KL forces small TV.
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 0.505384 ✓ checked
See also: total variation distance · Kullback–Leibler divergence
\mathrm{TV}(p,q)\le\sqrt{\tfrac12D_{\mathrm{KL}}(p\|q)}R(D) = ½ log₂(σ²/D)
Bits per sample needed to describe a Gaussian to squared error D.
Worked example: s2 = 4, D = 1 → 1 ✓ checked
See also: Shannon–Hartley capacity (AWGN)
R(D)=\tfrac12\log_2\frac{\sigma^2}{D}\quad(0<D\le\sigma^2)H_α = (1/(1 − α)) log Σ p(x)^α
where α ≥ 0, α ≠ 1
α → 1 gives Shannon, α = 2 collision entropy, α → ∞ min-entropy.
Worked example: p = [0.7, 0.3], a = 2 → 0.544727 ✓ checked
See also: Shannon entropy · Tsallis entropy · min-entropy
H_\alpha(X)=\frac1{1-\alpha}\log\sum_xp(x)^\alphaH(X) = −Σₓ p(x) log p(x)
where log base 2 gives bits, base e gives nats
Maximised by the uniform distribution (log n). Zero when the outcome is certain.
Worked example: p = [0.5, 0.5] → 0.693147 ✓ checked
See also: cross-entropy · binary entropy
H(X)=-\sum_x p(x)\log p(x)C = B log₂(1 + S/N)
where B = bandwidth in Hz, S/N = signal-to-noise power ratio
Per channel use: ½ log₂(1 + SNR).
Worked example: B = 1e+06, snr = 15 → 4e+06 ✓ checked
See also: capacity of a binary symmetric channel · signal-to-noise ratio in decibels
C=B\log_2\Big(1+\frac SN\Big)\ \text{bits/s}SNR_dB = 10 log₁₀(P_s/P_n) = 20 log₁₀(A_s/A_n)
where P = power, A = amplitude
Worked example: ps = 100, pn = 1 → 20 ✓ checked
See also: Shannon–Hartley capacity (AWGN) · peak signal-to-noise ratio
\mathrm{SNR}_{\mathrm{dB}}=10\log_{10}\frac{P_s}{P_n}=20\log_{10}\frac{A_s}{A_n}K = exp(−C/ε); u ← a/(Kv); v ← b/(Kᵀu); P = diag(u) K diag(v)
where a, b = marginals, C = cost matrix
The plan P has marginals a and b at convergence.
Worked example: a = [0.5, 0.5], b = [0.5, 0.5], C = [[0, 1], [1, 0]], eps = 0.5 → [0.5, 0.5] ✓ checked
See also: Wasserstein-1 distance (1-D)
K=e^{-C/\varepsilon},\quad u\leftarrow\frac{a}{Kv},\quad v\leftarrow\frac{b}{K^{\top}u},\quad P=\operatorname{diag}(u)K\operatorname{diag}(v)H(X) ≤ L̄ < H(X) + 1
where L̄ = expected code length of an optimal prefix code
Entropy is the compression limit. Equality when all probabilities are powers of ½.
Worked example: p = [0.5, 0.25, 0.125, 0.125], l = [1, 2, 3, 3] → [1.75, 1.75] ✓ checked
See also: Shannon entropy · Kraft inequality
H(X)\le\bar L<H(X)+1\ \text{bits}TV = ½ Σ |p − q|
The largest difference in probability of any event. Bounded in [0, 1].
Worked example: p = [0.5, 0.5], q = [0.9, 0.1] → 0.4 ✓ checked
See also: Kullback–Leibler divergence · Pinsker's inequality
\mathrm{TV}(p,q)=\tfrac12\sum_x|p(x)-q(x)|=\sup_A|p(A)-q(A)|S_q = (1 − Σ p(x)^q)/(q − 1)
Non-extensive; q → 1 gives Shannon (in nats).
Worked example: p = [0.7, 0.3], q = 2 → 0.42 ✓ checked
See also: Rényi entropy
S_q=\frac{1-\sum_xp(x)^q}{q-1}W₁ = ∫ |F_p − F_q| dx
The cost of moving mass; unlike KL it stays finite and informative for disjoint supports.
Worked example: p = [0.5, 0.5], q = [0.2, 0.8] → 0.3 ✓ checked
See also: Wasserstein-2 distance between Gaussians (1-D)
W_1(p,q)=\int|F_p(x)-F_q(x)|\,dxW₂² = (μ₁ − μ₂)² + (σ₁ − σ₂)²
The multivariate case adds a matrix-square-root term; used in the FID.
Worked example: m1 = 0, m2 = 1, s1 = 1, s2 = 2 → 2 ✓ checked
See also: Wasserstein-1 distance (1-D) · Fréchet inception distance
W_2^2\big(\mathcal N(\mu_1,\sigma_1^2),\mathcal N(\mu_2,\sigma_2^2)\big)=(\mu_1-\mu_2)^2+(\sigma_1-\sigma_2)^2R(θ) = [[cos θ, −sin θ], [sin θ, cos θ]]
where counter-clockwise by θ
Worked example: th = 1.5708 → [6.12323e-17, 1] ✓ checked
See also: orthogonal matrix · Rodrigues' rotation formula
R(\theta)=\begin{pmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{pmatrix}|x·y| ≤ ‖x‖ ‖y‖
Equality iff the vectors are parallel.
Worked example: x = [1, 2], y = [3, 4] → 0.18034 ✓ checked
See also: dot product · cosine similarity
|\mathbf x\cdot\mathbf y|\le\|\mathbf x\|\,\|\mathbf y\|A² − tr(A)A + det(A)I = 0 (2×2)
A matrix satisfies its own characteristic equation.
Worked example: A = [[1, 2], [3, 4]] → 4.44089e-16 ✓ checked
See also: eigenvalue equation · trace
p(A)=0,\quad p(\lambda)=\det(\lambda I-A);\ \ \text{for }2\times2:\ A^2-\operatorname{tr}(A)A+\det(A)I=0A = L Lᵀ
Twice as fast as LU for symmetric positive-definite systems; used to sample from N(μ, Σ).
Worked example: A = [[2, 1], [1, 3]] → 4.44089e-16 ✓ checked
See also: LU decomposition and solving · QR decomposition
A=LL^{\top}\quad(A\succ0,\ L\text{ lower triangular})κ(A) = σ_max / σ_min
Large κ: small input errors become large output errors; slows gradient descent.
Worked example: A = [[10, 0], [0, 2]] → 5 ✓ checked
See also: singular value decomposition · gradient descent rate (strongly convex)
\kappa(A)=\frac{\sigma_{\max}(A)}{\sigma_{\min}(A)}CG: αₖ = rᵀr/(pᵀAp); x += αp; r −= αAp; β = rₙₑwᵀrₙₑw/rᵀr; p = r + βp
where A symmetric positive definite
Solves an n×n system in at most n steps in exact arithmetic.
Worked example: A = [[2, 1], [1, 3]], b = [1, 2] → [0.2, 0.6] ✓ checked
See also: minimiser of a quadratic
\alpha_k=\frac{\mathbf r_k^{\top}\mathbf r_k}{\mathbf p_k^{\top}A\mathbf p_k},\ \mathbf x_{k+1}=\mathbf x_k+\alpha_k\mathbf p_k,\ \mathbf r_{k+1}=\mathbf r_k-\alpha_kA\mathbf p_k,\ \beta_k=\frac{\mathbf r_{k+1}^{\top}\mathbf r_{k+1}}{\mathbf r_k^{\top}\mathbf r_k},\ \mathbf p_{k+1}=\mathbf r_{k+1}+\beta_k\mathbf p_kcos θ = x·y / (‖x‖‖y‖)
Ignores magnitude; 1 = same direction, 0 = orthogonal.
Worked example: x = [1, 0], y = [1, 1] → 0.707107 ✓ checked
See also: dot product · InfoNCE (contrastive) loss
\cos\theta=\frac{\mathbf x\cdot\mathbf y}{\|\mathbf x\|\,\|\mathbf y\|}xᵢ = det(Aᵢ)/det(A)
where Aᵢ = A with column i replaced by b
Elegant, but far slower than elimination for large systems.
Worked example: A = [[2, 1], [1, 3]], b = [3, 5] → [0.8, 1.4] ✓ checked
See also: determinant (2×2)
x_i=\frac{\det A_i}{\det A}a × b = (a₂b₃ − a₃b₂, a₃b₁ − a₁b₃, a₁b₂ − a₂b₁)
The triple product a·(b×c) equals det[a b c] = the signed volume of the parallelepiped.
Worked example: a = [1, 0, 0], b = [0, 1, 0], c = [0, 0, 2] → [0, 0, 1, 2] ✓ checked
See also: dot product
\mathbf a\times\mathbf b=\begin{pmatrix}a_2b_3-a_3b_2\\ a_3b_1-a_1b_3\\ a_1b_2-a_2b_1\end{pmatrix},\quad \|\mathbf a\times\mathbf b\|=\|\mathbf a\|\|\mathbf b\|\sin\thetatr(AB) = tr(BA)
Lets you rearrange quadratic forms: xᵀAx = tr(A xxᵀ).
Worked example: A = [[1, 2], [3, 4]], B = [[0, 1], [1, 0]] → 0 ✓ checked
See also: trace
\operatorname{tr}(AB)=\operatorname{tr}(BA),\qquad \operatorname{tr}(ABC)=\operatorname{tr}(CAB)d(A⁻¹) = −A⁻¹ (dA) A⁻¹
Worked example: A = [[2, 1], [0, 1]], dA = [[1, 0], [0, 2]] → [[-0.25, 1.25], [0, -2]] ✓ checked
See also: gradient of log-determinant
\frac{d}{dt}A(t)^{-1}=-A^{-1}\frac{dA}{dt}A^{-1}det = ad − bc
Zero iff the matrix is singular.
Worked example: A = [[1, 2], [3, 4]] → -2 ✓ checked
See also: inverse (2×2)
\det\begin{pmatrix}a&b\\c&d\end{pmatrix}=ad-bcdet = aei + bfg + cdh − ceg − bdi − afh
Worked example: M = [9 values] → 0 ✓ checked
See also: determinant (2×2)
\det\begin{pmatrix}a&b&c\\d&e&f\\g&h&i\end{pmatrix}=aei+bfg+cdh-ceg-bdi-afhdet A = Π λᵢ; tr A = Σ λᵢ
Worked example: A = [[2, 1], [1, 3]] → [5, 5] ✓ checked
See also: trace · eigenvalue equation
\det A=\prod_i\lambda_i,\qquad \operatorname{tr}A=\sum_i\lambda_idet(A ⊗ B) = det(A)ᵐ det(B)ⁿ
Worked example: A = [[2, 0], [0, 3]], B = [[1, 1], [0, 2]] → 144 ✓ checked
See also: Kronecker product
\det(A\otimes B)=\det(A)^{m}\det(B)^{n}\quad(A\in\mathbb R^{n\times n},\ B\in\mathbb R^{m\times m})det(AB) = det A det B; det(A⁻¹) = 1/det A; det(cA) = cⁿ det A
Worked example: A = [[1, 2], [3, 4]], B = [[0, 1], [1, 1]] → [2, -0.5, -18] ✓ checked
See also: determinant (2×2)
\det(AB)=\det A\det B,\quad \det A^{\top}=\det A,\quad \det A^{-1}=\frac1{\det A},\quad \det(cA)=c^n\det Ax·y = Σ xᵢyᵢ = ‖x‖‖y‖ cos θ
where θ = angle between the vectors
Zero iff orthogonal.
Worked example: x = [1, 2, 3], y = [4, 5, 6] → 32 ✓ checked
See also: cosine similarity
\mathbf{x}\cdot\mathbf{y}=\sum_i x_i y_i=\|\mathbf x\|\|\mathbf y\|\cos\theta‖A − A_k‖_F² = Σ_{i>k} σᵢ²The truncated SVD is the best rank-k approximation in both Frobenius and spectral norm.
Worked example: A = [[3, 0], [0, 1]], k = 1 → 1 ✓ checked
See also: singular value decomposition · principal component analysis
\min_{\operatorname{rank}B\le k}\|A-B\|_F^2=\sum_{i>k}\sigma_i^2,\quad B=U_k\Sigma_kV_k^{\top}A v = λ v
where v ≠ 0 eigenvector, λ eigenvalue
Eigenvalues solve det(A − λI) = 0.
Worked example: A = [[2, 0], [0, 3]], v = [1, 0], lam = 2 → 0 ✓ checked
See also: trace · singular value decomposition · principal component analysis
A\mathbf v=\lambda\mathbf v‖x‖₂ = √(Σ xᵢ²)
Worked example: x = [3, 4] → 5 ✓ checked
See also: L1 norm (Manhattan) · Lp norm
\|\mathbf x\|_2=\sqrt{\sum_i x_i^2}EVR_k = λ_k / Σⱼ λⱼ
Worked example: lam = [4, 1] → 0.8 ✓ checked
See also: principal component analysis
\mathrm{EVR}_k=\frac{\lambda_k}{\sum_j\lambda_j}‖A‖_F = √(Σᵢⱼ Aᵢⱼ²)
Worked example: A = [[1, 2], [3, 4]] → 5.47723 ✓ checked
See also: spectral norm
\|A\|_F=\sqrt{\sum_{i,j}A_{ij}^2}=\sqrt{\operatorname{tr}(A^\top A)}every eigenvalue lies in a disc centred at aᵢᵢ with radius Σⱼ≠ᵢ |aᵢⱼ|
A quick bound on the spectrum; strict diagonal dominance implies invertibility.
Worked example: A = [[4, 1], [1, 2]] → 1 ✓ checked
See also: eigenvalue equation
\lambda\in\bigcup_i\Big\{z:\ |z-a_{ii}|\le\sum_{j\ne i}|a_{ij}|\Big\}∇ₓ aᵀx = a
Worked example: a = [1, -2], x = [0.5, 0.5] → [1, -2] ✓ checked
See also: gradient of a quadratic form
\nabla_{\mathbf x}\,\mathbf a^{\top}\mathbf x=\mathbf a∇ₓ xᵀAx = (A + Aᵀ)x
2Ax when A is symmetric.
Worked example: A = [[1, 2], [0, 3]], x = [1, 2] → [6, 14] ✓ checked
See also: gradient of a linear form · Hessian of a quadratic
\nabla_{\mathbf x}\,\mathbf x^{\top}A\mathbf x=(A+A^{\top})\mathbf x∂/∂X tr(AX) = Aᵀ; ∂/∂X tr(XᵀAX) = (A + Aᵀ)X
Worked example: A = [[1, 2], [3, 4]], X = [[1, 0], [2, 1]] → [[1, 3], [2, 4]] ✓ checked
See also: gradient of a quadratic form
\frac{\partial}{\partial X}\operatorname{tr}(AX)=A^{\top},\qquad \frac{\partial}{\partial X}\operatorname{tr}(X^{\top}AX)=(A+A^{\top})X∂/∂X ln det X = X⁻ᵀ
Appears in Gaussian log-likelihoods.
Worked example: X = [[2, 1], [0, 1]] → [[0.5, 0], [-0.5, 1]] ✓ checked
See also: multivariate normal density
\frac{\partial}{\partial X}\ln\det X=X^{-\top}G = XᵀX; Gᵢⱼ = xᵢ·xⱼ
Positive semi-definite. Kernel methods work with G (or K) instead of X.
Worked example: X = [[1, 2], [3, 4], [5, 6]] → 0.264505 ✓ checked
See also: RBF (Gaussian) kernel
G=X^{\top}X,\quad G_{ij}=\mathbf x_i\cdot\mathbf x_j,\quad G\succeq0subtract the projections onto earlier vectors, then normalise
Classical Gram–Schmidt is unstable; use modified Gram–Schmidt or Householder QR.
Worked example: v1 = [1, 1], v2 = [1, 0] → 2.77556e-16 ✓ checked
See also: projection onto a vector · QR decomposition
\mathbf u_k=\mathbf v_k-\sum_{j<k}\operatorname{proj}_{\mathbf u_j}(\mathbf v_k),\qquad \mathbf e_k=\frac{\mathbf u_k}{\|\mathbf u_k\|}L = D − A; xᵀLx = Σ over edges (xᵢ − xⱼ)²
where D = degree matrix, A = adjacency
Eigenvalue 0 for each connected component; the second-smallest eigenvalue (Fiedler value) measures connectivity.
Worked example: A = [9 values] → [3.96029e-17, 1, 3] ✓ checked
See also: normalised graph Laplacian
L=D-A,\qquad \mathbf x^{\top}L\mathbf x=\sum_{(i,j)\in E}(x_i-x_j)^2(A ∘ B)ᵢⱼ = Aᵢⱼ Bᵢⱼ
Schur product theorem: the Hadamard product of two PSD matrices is PSD. Gating in LSTMs and attention masks use it.
Worked example: A = [[1, 2], [3, 4]], B = [[0, 1], [1, 2]] → [[0, 2], [3, 8]] ✓ checked
See also: matrix product (entry)
(A\circ B)_{ij}=A_{ij}B_{ij}∇²(½xᵀAx − bᵀx) = A for symmetric A
Convex iff A is positive semi-definite.
Worked example: A = [[2, 1], [1, 3]], b = [1, 1], x = [0.3, -0.2] → [[2, 1], [1, 3]] ✓ checked
See also: gradient of a quadratic form
\nabla^2\Big(\tfrac12\mathbf x^{\top}A\mathbf x-\mathbf b^{\top}\mathbf x\Big)=\tfrac12(A+A^{\top})H = I − 2vvᵀ/(vᵀv)
Orthogonal and symmetric; used in QR to zero out columns.
Worked example: v = [1, 0], x = [3, 4] → [-3, 4] ✓ checked
See also: QR decomposition
H=I-2\frac{\mathbf v\mathbf v^{\top}}{\mathbf v^{\top}\mathbf v}‖A‖₁ = max column sum; ‖A‖∞ = max row sum
Worked example: A = [[1, -2], [3, 4]] → [6, 7] ✓ checked
See also: spectral norm
\|A\|_1=\max_j\sum_i|a_{ij}|,\qquad \|A\|_\infty=\max_i\sum_j|a_{ij}|A⁻¹ = (1/(ad−bc)) · [[d, −b], [−c, a]]
Prefer solving linear systems to forming an inverse numerically.
Worked example: A = [[1, 2], [3, 4]] → [[-2, 1], [1.5, -0.5]] ✓ checked
See also: determinant (2×2)
\begin{pmatrix}a&b\\c&d\end{pmatrix}^{-1}=\frac{1}{ad-bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}xᵢ ← (bᵢ − Σⱼ≠ᵢ aᵢⱼxⱼ)/aᵢᵢ
Guaranteed for strictly diagonally dominant matrices.
Worked example: A = [[4, 1], [2, 5]], b = [9, 12] → [1.83333, 1.66667] ✓ checked
See also: conjugate gradient method · LU decomposition and solving
x^{(k+1)}_i=\frac1{a_{ii}}\Big(b_i-\sum_{j\ne i}a_{ij}x^{(k)}_j\Big);\qquad \text{converges iff }\rho(M)<1(A ⊗ B)(C ⊗ D) = AC ⊗ BD
(A ⊗ B)⁻¹ = A⁻¹ ⊗ B⁻¹; eigenvalues are products λᵢ(A)μⱼ(B).
Worked example: A = [[1, 2], [0, 1]], B = [[1, 0], [1, 1]], C = [[0, 1], [1, 0]], D = [[2, 0], [0, 1]] → 0 ✓ checked
See also: vec identity · determinant of a Kronecker product
A\otimes B=\begin{pmatrix}a_{11}B&\cdots&a_{1n}B\\ \vdots&\ddots&\vdots\\ a_{m1}B&\cdots&a_{mn}B\end{pmatrix},\qquad (A\otimes B)(C\otimes D)=AC\otimes BD‖x‖₁ = Σ |xᵢ|
Promotes sparsity when used as a penalty.
Worked example: x = [1, -2, 3] → 6 ✓ checked
See also: Euclidean (L2) norm · L1 penalty (lasso)
\|\mathbf x\|_1=\sum_i |x_i|Rβ = Qᵀy
where X = QR
Avoids squaring the condition number as the normal equations do.
Worked example: X = [[1, 0], [1, 1], [1, 2]], y = [1, 3, 5] → [1, 2] ✓ checked
See also: ordinary least squares (normal equations) · QR decomposition
\min_\beta\|X\beta-\mathbf y\|\ \Rightarrow\ R\beta=Q^{\top}\mathbf y‖x‖ₚ = ( Σ |xᵢ|ᵖ )^(1/p)
where p ≥ 1
p → ∞ gives the max norm.
Worked example: x = [1, 2, 2], p = 3 → 2.57128 ✓ checked
See also: L1 norm (Manhattan) · Euclidean (L2) norm · max (L∞) norm
\|\mathbf x\|_p=\Big(\sum_i |x_i|^p\Big)^{1/p}PA = LU; solve Ly = Pb, then Ux = y
O(n³) once, then O(n²) per right-hand side.
Worked example: A = [[2, 1], [1, 3]], b = [3, 5] → [0.8, 1.4] ✓ checked
See also: Cholesky decomposition
PA=LU,\quad A\mathbf x=\mathbf b\ \Rightarrow\ L\mathbf y=P\mathbf b,\ U\mathbf x=\mathbf yd_M = √((x − μ)ᵀ Σ⁻¹ (x − μ))
Distance in units of standard deviation, accounting for correlations. Σ = I gives the Euclidean distance.
Worked example: x = [2, 1], mu = [0, 0], S = [[4, 0], [0, 1]] → 1.41421 ✓ checked
See also: multivariate normal density · Euclidean (L2) norm
d_M(\mathbf x,\boldsymbol\mu)=\sqrt{(\mathbf x-\boldsymbol\mu)^{\top}\Sigma^{-1}(\mathbf x-\boldsymbol\mu)}det(A + uvᵀ) = (1 + vᵀA⁻¹u) det A
Worked example: A = [[2, 0], [0, 3]], u = [1, 1], v = [1, 0] → 9 ✓ checked
See also: Sherman–Morrison formula
\det(A+\mathbf u\mathbf v^{\top})=\big(1+\mathbf v^{\top}A^{-1}\mathbf u\big)\det Ae^A = Σ Aᵏ/k!
Solves x′ = Ax: x(t) = e^(tA) x₀.
Worked example: A = [[1, 0], [0, 2]] → [2.71828, 7.38906] ✓ checked
See also: matrix power via eigendecomposition
e^{A}=\sum_{k=0}^{\infty}\frac{A^k}{k!},\qquad \frac{d}{dt}e^{tA}=Ae^{tA}Aᵏ = QΛᵏQᵀ
Also f(A) = Q f(Λ) Qᵀ for any function f.
Worked example: A = [[2, 1], [1, 3]], k = 3 → 7.10543e-15 ✓ checked
See also: spectral decomposition (symmetric) · matrix exponential
A^{k}=Q\Lambda^{k}Q^{\top}(AB)ᵢⱼ = Σₖ Aᵢₖ Bₖⱼ
where A is m×n, B is n×p, AB is m×p
Not commutative in general.
Worked example: A = [[1, 2], [3, 4]], B = [[5, 6], [7, 8]] → 22 ✓ checked
See also: dot product
(AB)_{ij}=\sum_k A_{ik}B_{kj}‖x‖∞ = maxᵢ |xᵢ|
Worked example: x = [1, -7, 3] → 7 ✓ checked
See also: Lp norm
\|\mathbf x\|_\infty=\max_i |x_i|x* = A⁻¹b; f(x*) = −½ bᵀA⁻¹b
where A ≻ 0
Worked example: A = [[2, 1], [1, 3]], b = [1, 2] → [0.2, 0.6, -0.7] ✓ checked
See also: Hessian of a quadratic · conjugate gradient method
\min_{\mathbf x}\tfrac12\mathbf x^{\top}A\mathbf x-\mathbf b^{\top}\mathbf x\ \Rightarrow\ \mathbf x^*=A^{-1}\mathbf b,\quad f(\mathbf x^*)=-\tfrac12\mathbf b^{\top}A^{-1}\mathbf bA⁺ = VΣ⁺Uᵀ = (AᵀA)⁻¹Aᵀ
Gives the minimum-norm least-squares solution x = A⁺b.
Worked example: A = [[1, 0], [1, 1], [1, 2]] → 3.33067e-16 ✓ checked
See also: ordinary least squares (normal equations) · singular value decomposition
A^{+}=V\Sigma^{+}U^{\top}=(A^{\top}A)^{-1}A^{\top}\ (\text{full column rank})L_sym = I − D^(−1/2) A D^(−1/2)
Eigenvalues in [0, 2]. The GCN propagation matrix is a renormalised version.
Worked example: A = [9 values] → [-1.01113e-16, 1, 2] ✓ checked
See also: graph Laplacian · graph convolution (GCN)
\mathcal L=I-D^{-1/2}AD^{-1/2}‖A‖_* = Σ σᵢ
The convex surrogate for rank; used in matrix completion.
Worked example: A = [[3, 0], [0, 1]] → 4 ✓ checked
See also: spectral norm · Frobenius norm
\|A\|_*=\sum_i\sigma_i(A)β̂ = (XᵀX)⁻¹ Xᵀ y
where X = design matrix, y = targets
Minimises ‖y − Xβ‖². Solve the system; do not invert.
Worked example: X = [[1, 0], [1, 1], [1, 2]], y = [1, 3, 5] → [1, 2] ✓ checked
See also: ridge regression (closed form) · linear regression model
\hat\beta=(X^{\top}X)^{-1}X^{\top}yQᵀQ = I; det Q = ±1; ‖Qx‖ = ‖x‖
Preserves lengths and angles; all singular values equal 1, so it neither explodes nor vanishes gradients.
Worked example: Q = [[0.6, -0.8], [0.8, 0.6]], x = [3, 4] → 5 ✓ checked
See also: 2-D rotation matrix
Q^{\top}Q=QQ^{\top}=I,\quad \det Q=\pm1,\quad \|Q\mathbf x\|=\|\mathbf x\|R = UVᵀ where BAᵀ = UΣVᵀ
The best rotation aligning one point set to another.
Worked example: A = [[1, 0], [0, 1]], B = [[0, -1], [1, 0]] → 0 ✓ checked
See also: singular value decomposition · orthogonal matrix
\min_{R^{\top}R=I}\|RA-B\|_F\ \Rightarrow\ R=UV^{\top},\quad BA^{\top}=U\Sigma V^{\top}P = A(AᵀA)⁻¹Aᵀ; P² = P = Pᵀ
Projects onto the column space of A; I − P projects onto its orthogonal complement.
Worked example: A = [[1, 0], [1, 1], [1, 2]] → 1.11022e-16 ✓ checked
See also: hat matrix and leverage · projection onto a vector
P=A(A^{\top}A)^{-1}A^{\top},\quad P^2=P=P^{\top}r = d M r + (1 − d)/N · 1
where M = column-stochastic link matrix, d ≈ 0.85
The stationary distribution of a random surfer who sometimes teleports.
Worked example: M = [[0, 1], [1, 0]], d = 0.85 → [0.5, 0.5] ✓ checked
See also: stationary distribution of a Markov chain · power iteration
\mathbf r=d\,M\mathbf r+\frac{1-d}{N}\mathbf 1‖x + y‖² + ‖x − y‖² = 2‖x‖² + 2‖y‖²
Holds exactly when the norm comes from an inner product.
Worked example: x = [1, 2], y = [3, -1] → 30 ✓ checked
See also: Euclidean (L2) norm
\|\mathbf x+\mathbf y\|^2+\|\mathbf x-\mathbf y\|^2=2\|\mathbf x\|^2+2\|\mathbf y\|^2Σ ⟨x, eᵢ⟩² = ‖x‖²
Energy is preserved by an orthonormal change of basis (and by the unitary DFT).
Worked example: x = [1, 2, 2], Q = [9 values] → 9 ✓ checked
See also: Parseval's theorem (DFT)
\sum_i\langle\mathbf x,\mathbf e_i\rangle^2=\|\mathbf x\|^2positive matrix ⇒ spectral radius is a simple, positive eigenvalue with a positive eigenvector
Row-stochastic matrices have ρ = 1.
Worked example: P = [[0.9, 0.1], [0.5, 0.5]] → 1 ✓ checked
See also: stationary distribution of a Markov chain
A>0\ \Rightarrow\ \rho(A)=\lambda_1>0\ \text{simple, with a positive eigenvector}vₖ₊₁ = Avₖ/‖Avₖ‖
Converges at rate |λ₂/λ₁|. The core of PageRank.
Worked example: A = [[2, 0], [0, 1]] → 2 ✓ checked
See also: Rayleigh quotient · PageRank
\mathbf v_{k+1}=\frac{A\mathbf v_k}{\|A\mathbf v_k\|}\ \to\ \text{dominant eigenvector}max wᵀΣw s.t. ‖w‖ = 1 → w = top eigenvector of Σ
where Σ = covariance matrix
The maximum equals the largest eigenvalue: the variance captured by that direction.
Worked example: S = [[2, 0], [0, 1]] → 2 ✓ checked
See also: eigenvalue equation · singular value decomposition · explained variance ratio
\max_{\|\mathbf w\|=1}\ \mathbf w^{\top}\Sigma\,\mathbf w\ \Rightarrow\ \mathbf w=\text{top eigenvector of }\Sigmaproj_u(v) = (v·u / u·u) u
Worked example: v = [2, 3], u = [1, 0] → [2, 0] ✓ checked
See also: dot product
\operatorname{proj}_{\mathbf u}(\mathbf v)=\frac{\mathbf v\cdot\mathbf u}{\mathbf u\cdot\mathbf u}\,\mathbf ux ⊥ y ⇒ ‖x + y‖² = ‖x‖² + ‖y‖²
The reason uncorrelated variances add.
Worked example: x = [3, 0], y = [0, 4] → 25 ✓ checked
See also: dot product
\mathbf x\perp\mathbf y\ \Rightarrow\ \|\mathbf x+\mathbf y\|^2=\|\mathbf x\|^2+\|\mathbf y\|^2A = QR
The stable way to solve least squares: Rβ = Qᵀy.
Worked example: A = [[1, 2], [3, 4], [5, 6]] → 8.88178e-16 ✓ checked
See also: Cholesky decomposition · least squares via QR
A=QR\quad(Q^{\top}Q=I,\ R\text{ upper triangular})rank(A) + nullity(A) = n
where n = number of columns
Worked example: A = [[1, 2, 3], [2, 4, 6]] → [1, 2] ✓ checked
See also: singular value decomposition
\operatorname{rank}(A)+\operatorname{nullity}(A)=nR(x) = xᵀAx / xᵀx
where A symmetric
Its extremes are the extreme eigenvalues; the objective behind PCA.
Worked example: A = [[2, 0], [0, 1]], x = [1, 1] → 1.5 ✓ checked
See also: principal component analysis · power iteration
R(\mathbf x)=\frac{\mathbf x^{\top}A\mathbf x}{\mathbf x^{\top}\mathbf x},\quad \lambda_{\min}\le R(\mathbf x)\le\lambda_{\max}β̂ = (XᵀX + λI)⁻¹ Xᵀ y
where λ ≥ 0 = penalty strength
Shrinks coefficients; always invertible for λ > 0.
Worked example: X = [[1, 0], [1, 1], [1, 2]], y = [1, 3, 5], lam = 1 → [1, 1.66667] ✓ checked
See also: ordinary least squares (normal equations) · L2 penalty (weight decay)
\hat\beta=(X^{\top}X+\lambda I)^{-1}X^{\top}yR = I + sin θ K + (1 − cos θ) K²
where k = unit axis, K = its cross-product matrix
Rotation by θ about an axis in 3-D.
Worked example: th = 1.5708 → [1.11022e-16, 1, 0] ✓ checked
See also: 2-D rotation matrix · cross product and triple product
R=I+\sin\theta\,K+(1-\cos\theta)K^2,\quad K=[\mathbf k]_\times‖A‖_Sp = (Σ σᵢᵖ)^(1/p)
p = 1 nuclear, p = 2 Frobenius, p = ∞ spectral.
Worked example: s = [3, 1], p = 3 → 3.03659 ✓ checked
See also: nuclear norm
\|A\|_{S_p}=\Big(\sum_i\sigma_i^p\Big)^{1/p}det[[A, B], [C, D]] = det(A) det(D − CA⁻¹B)
where D − CA⁻¹B = Schur complement of A
Also appears in block matrix inversion and Gaussian conditioning.
Worked example: A = [[2]], B = [[1]], C = [[1]], D = [[3]] → 8.88178e-16 ✓ checked
See also: multivariate normal: conditional distribution
\det\begin{pmatrix}A&B\\C&D\end{pmatrix}=\det(A)\,\det\big(D-CA^{-1}B\big)(A + uvᵀ)⁻¹ = A⁻¹ − A⁻¹uvᵀA⁻¹ / (1 + vᵀA⁻¹u)
A rank-one update of an inverse in O(n²): the heart of recursive least squares.
Worked example: A = [[2, 0], [0, 3]], u = [1, 1], v = [1, 0] → 5.55112e-17 ✓ checked
See also: Woodbury matrix identity
(A+\mathbf u\mathbf v^{\top})^{-1}=A^{-1}-\frac{A^{-1}\mathbf u\mathbf v^{\top}A^{-1}}{1+\mathbf v^{\top}A^{-1}\mathbf u}A = U Σ Vᵀ
where U, V orthogonal; Σ diagonal with σ₁ ≥ σ₂ ≥ … ≥ 0
Best rank-k approximation (Eckart–Young): keep the top k singular values.
Worked example: A = [[1, 2], [3, 4]] → 8.88178e-16 ✓ checked
See also: spectral norm · principal component analysis
A=U\Sigma V^{\top}A = QΛQᵀ
Real eigenvalues and orthogonal eigenvectors. A is positive definite iff all λ > 0.
Worked example: A = [[2, 1], [1, 3]] → 6.66134e-16 ✓ checked
See also: eigenvalue equation · singular value decomposition
A=Q\Lambda Q^{\top},\quad Q^{\top}Q=I,\ \Lambda=\operatorname{diag}(\lambda_i)‖A‖₂ = σ_max(A) = √λ_max(AᵀA)
The largest factor by which A stretches a vector. Bounds the Lipschitz constant of a linear layer.
Worked example: A = [[3, 0], [0, 1]] → 3 ✓ checked
See also: singular value decomposition · Frobenius norm
\|A\|_2=\sigma_{\max}(A)=\sqrt{\lambda_{\max}(A^\top A)}πP = π
where P = row-stochastic transition matrix
The left eigenvector for eigenvalue 1. For a two-state chain π = (b, a)/(a + b) with switch rates a, b.
Worked example: P = [[0.9, 0.1], [0.5, 0.5]] → [0.833333, 0.166667] ✓ checked
See also: PageRank · entropy rate of a Markov chain
\boldsymbol\pi P=\boldsymbol\pi,\quad \sum_i\pi_i=1‖AB‖ ≤ ‖A‖ ‖B‖
Why the product of layers' spectral norms bounds a network's Lipschitz constant.
Worked example: A = [[1, 2], [0, 1]], B = [[1, 0], [1, 1]] → 0.0419511 ✓ checked
See also: spectral norm · Lipschitz continuity
\|AB\|\le\|A\|\,\|B\|tr(A) = Σ Aᵢᵢ = Σ λᵢ
where λᵢ = eigenvalues
Cyclic: tr(ABC) = tr(BCA).
Worked example: A = [[1, 2], [3, 4]] → 5 ✓ checked
See also: eigenvalue equation
\operatorname{tr}(A)=\sum_i A_{ii}=\sum_i\lambda_ivec(AXB) = (Bᵀ ⊗ A) vec(X)
where vec stacks columns
Turns a matrix equation into an ordinary linear system.
Worked example: A = [[1, 2], [3, 4]], X = [[1, 0], [2, 1]], B = [[0, 1], [1, 1]] → 0 ✓ checked
See also: Kronecker product
\operatorname{vec}(AXB)=\big(B^{\top}\otimes A\big)\operatorname{vec}(X)|λᵢ(A + E) − λᵢ(A)| ≤ ‖E‖₂
where A, E symmetric
Eigenvalues of symmetric matrices are well-conditioned.
Worked example: A = [[2, 0], [0, 1]], E = [[0.1, 0], [0, -0.05]] → -8.32667e-17 ✓ checked
See also: spectral norm
|\lambda_i(A+E)-\lambda_i(A)|\le\|E\|_2(A + UCV)⁻¹ = A⁻¹ − A⁻¹U(C⁻¹ + VA⁻¹U)⁻¹VA⁻¹
Cheap when the update has low rank k ≪ n. Used in Gaussian processes and Kalman filters.
Worked example: A = [9 values], U = [[1], [0], [1]], C = [[2]], V = [[1, 1, 0]] → 0 ✓ checked
See also: Sherman–Morrison formula
(A+UCV)^{-1}=A^{-1}-A^{-1}U\big(C^{-1}+VA^{-1}U\big)^{-1}VA^{-1}HM ≤ GM ≤ AM ≤ QM (quadratic mean)
Worked example: x = [1, 4] → [1.6, 2, 2.5, 2.91548] ✓ checked
See also: AM–GM inequality · geometric mean · harmonic mean
\min\le\mathrm{HM}\le\mathrm{GM}\le\mathrm{AM}\le\mathrm{QM}\le\maxΣ (a + kd) = (n/2)(2a + (n − 1)d)
Gauss: 1 + 2 + … + n = n(n + 1)/2.
Worked example: a = 3, d = 2, n = 10 → 120 ✓ checked
See also: sums of powers of integers
\sum_{k=0}^{n-1}(a+kd)=\frac n2\big(2a+(n-1)d\big)Σ 1/n² = π²/6; Σ (−1)ⁿ/(2n + 1) = π/4
Worked example: N = 200000 → [1.64493, 0.785397] ✓ checked
See also: harmonic numbers
\sum_{n=1}^\infty\frac1{n^2}=\frac{\pi^2}6,\qquad \sum_{n=0}^\infty\frac{(-1)^n}{2n+1}=\frac\pi4(1 + x)ⁿ ≥ 1 + nx
Worked example: x = 0.1, n = 5 → 0.11051 ✓ checked
See also: binomial theorem
(1+x)^n\ge1+nx\quad(x\ge-1,\ n\ge1)B(x, y) = Γ(x)Γ(y)/Γ(x + y)
Worked example: x = 2, y = 3 → 0.0833333 ✓ checked
See also: gamma function · beta distribution
B(x,y)=\int_0^1t^{x-1}(1-t)^{y-1}dt=\frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}bisection halves the bracket; secant uses a finite-difference slope
Bisection gains one bit per step; secant converges superlinearly (order ≈ 1.618).
Worked example: a = 1, b = 2 → 1.41421 ✓ checked
See also: Newton's method (root finding)
c=\frac{a+b}2;\qquad x_{n+1}=x_n-f(x_n)\frac{x_n-x_{n-1}}{f(x_n)-f(x_{n-1})}f(a) = (1/2πi) ∮ f(z)/(z − a) dz
Worked example: a = 0 → 1 ✓ checked
See also: Euler's formula and identity
f(a)=\frac1{2\pi i}\oint\frac{f(z)}{z-a}dz,\qquad \oint f=2\pi i\sum\operatorname{Res}|∫fg| ≤ ‖f‖_p ‖g‖_q, 1/p + 1/q = 1
For expectations: |E[XY]| ≤ √(E[X²]E[Y²]).
Worked example: b = 1 → 0.0648427 ✓ checked
See also: Cauchy–Schwarz inequality · Minkowski inequality (triangle inequality in Lp)
\Big|\int fg\Big|\le\|f\|_p\|g\|_q,\ \ \frac1p+\frac1q=1;\qquad p=q=2:\ \Big(\int fg\Big)^2\le\int f^2\int g^2d/dx f(g(x)) = f′(g(x)) · g′(x)
Backpropagation is the chain rule applied through a computation graph.
Worked example: x = 1 → 1.0806 ✓ checked
See also: backpropagation (layer error) · Taylor series
\frac{d}{dx}f(g(x))=f'(g(x))\,g'(x)∫ f(x) dx = ∫ f(g(u)) |det J_g(u)| du
where J_g = Jacobian matrix of g
Basis of normalising flows: log p(x) = log p(u) − log|det J|. Polar coordinates: dx dy = r dr dθ.
Worked example: R = 2 → 12.5664 ✓ checked
See also: substitution (change of variables) · normalising flow change of variables
\int_{\Omega}f(\mathbf x)\,d\mathbf x=\int_{\Omega'}f(g(\mathbf u))\,|\det J_g(\mathbf u)|\,d\mathbf usin x/x → 1; (1 + 1/n)ⁿ → e; (eˣ − 1)/x → 1
Worked example: x = 1e-05, n = 1e+07 → [1, 2.71828, 1.00001] ✓ checked
See also: L'Hôpital's rule
\lim_{x\to0}\frac{\sin x}{x}=1,\quad \lim_{n\to\infty}\Big(1+\frac1n\Big)^n=e,\quad \lim_{x\to0}\frac{e^x-1}{x}=1yₙ₊₁ = yₙ + (h/6)(k₁ + 2k₂ + 2k₃ + k₄)
Fourth-order accurate: global error O(h⁴).
Worked example: h = 0.1 → 2.71828 ✓ checked
See also: Euler's method
k_1=f(t,y),\ k_2=f(t+\tfrac h2,y+\tfrac h2k_1),\ k_3=f(t+\tfrac h2,y+\tfrac h2k_2),\ k_4=f(t+h,y+hk_3),\ y_{n+1}=y_n+\tfrac h6(k_1+2k_2+2k_3+k_4)(cos θ + i sin θ)ⁿ = cos nθ + i sin nθ
Worked example: th = 0.7, n = 5 → 4.00297e-16 ✓ checked
See also: Euler's formula and identity
(\cos\theta+i\sin\theta)^n=\cos n\theta+i\sin n\theta,\qquad z^n=1\Rightarrow z=e^{2\pi ik/n}(eˣ)′ = eˣ; (aˣ)′ = aˣ ln a; (ln x)′ = 1/x
Worked example: x = 2, a = 3 → [9.88751, 0.5] ✓ checked
See also: chain rule
\frac d{dx}e^{x}=e^x,\quad \frac d{dx}a^x=a^x\ln a,\quad \frac d{dx}\ln x=\frac1x,\quad \frac d{dx}\log_ax=\frac1{x\ln a}(sinh)′ = cosh; (cosh)′ = sinh; (tanh)′ = 1 − tanh²
The tanh derivative is why tanh activations saturate.
Worked example: x = 0.7 → [1.25517, 0.758584, 0.63474] ✓ checked
See also: hyperbolic tangent
(\sinh x)'=\cosh x,\quad (\cosh x)'=\sinh x,\quad (\tanh x)'=1-\tanh^2x=\operatorname{sech}^2xσ′ = σ(1 − σ); softplus′ = σ
Worked example: x = 0.4 → [0.240261, 0.598688] ✓ checked
See also: sigmoid (logistic function) · softplus
\sigma'(x)=\sigma(x)(1-\sigma(x)),\qquad \operatorname{softplus}'(x)=\sigma(x)(sin x)′ = cos x; (cos x)′ = −sin x; (tan x)′ = sec² x; (arctan x)′ = 1/(1 + x²)
Worked example: x = 0.5 → [0.877583, -0.479426, 1.29845, 1.1547, 0.8] ✓ checked
See also: chain rule
(\sin x)'=\cos x,\ (\cos x)'=-\sin x,\ (\tan x)'=\sec^2x,\ (\arcsin x)'=\frac1{\sqrt{1-x^2}},\ (\arctan x)'=\frac1{1+x^2}D_u f = ∇f · u
Steepest ascent is along ∇f with rate ‖∇f‖.
Worked example: g = [3, 4], u = [0.6, 0.8] → 5 ✓ checked
See also: gradient, divergence and curl
D_{\mathbf u}f=\nabla f\cdot\mathbf u,\quad \|\mathbf u\|=1∭ ∇·F dV = ∯ F·dS; ∬ (∇×F)·dS = ∮ F·dr
Green's theorem is the planar case.
Worked example: R = 1 → 3.14159 ✓ checked
See also: gradient, divergence and curl
\iiint_V\nabla\!\cdot\mathbf F\,dV=\oint_{\partial V}\mathbf F\cdot d\mathbf S,\qquad \iint_S(\nabla\times\mathbf F)\cdot d\mathbf S=\oint_{\partial S}\mathbf F\cdot d\mathbf rerf(x) = (2/√π) ∫₀ˣ e^(−t²) dt
Worked example: x = 1 → 0.842701 ✓ checked
See also: normal cumulative distribution
\operatorname{erf}(x)=\frac2{\sqrt\pi}\int_0^xe^{-t^2}dt,\qquad \Phi(x)=\tfrac12\big[1+\operatorname{erf}(x/\sqrt2)\big]yₙ₊₁ = yₙ + h f(tₙ, yₙ)
First-order accurate: global error O(h).
Worked example: h = 0.1 → 2.59374 ✓ checked
See also: classical Runge–Kutta (RK4)
y_{n+1}=y_n+h\,f(t_n,y_n)Σ f(k) = ∫ f + (f(a) + f(b))/2 + (1/12)(f′(b) − f′(a)) − …
Links sums and integrals; the trapezoid rule with correction terms from Bernoulli numbers. The series is asymptotic, so start the sum where f is smooth.
Worked example: a = 10, b = 100 → 0.0952165 ✓ checked
See also: trapezoid and Simpson's rules
\sum_{k=a}^{b}f(k)=\int_a^bf(x)\,dx+\frac{f(a)+f(b)}2+\frac1{12}\big(f^{(1)}(b)-f^{(1)}(a)\big)-\frac1{720}\big(f^{(3)}(b)-f^{(3)}(a)\big)+\cdotsforward, central and second differences
Gradient checking in deep learning uses the central difference with h ≈ 1e-5.
Worked example: x = 1, h = 0.0001 → [2.71828, 2.71828] ✓ checked
See also: Richardson extrapolation
f'(x)\approx\frac{f(x+h)-f(x)}h\ (O(h)),\quad \frac{f(x+h)-f(x-h)}{2h}\ (O(h^2)),\quad f''(x)\approx\frac{f(x+h)-2f(x)+f(x-h)}{h^2}integrating factor μ = e^(∫p dx)
Worked example: k = -0.5, y0 = 2, t = 2 → 0.735757 ✓ checked
See also: second-order linear ODE with constant coefficients
y'+p(x)y=q(x)\ \Rightarrow\ y=\frac1\mu\Big(\int\mu q\,dx+C\Big),\ \ \mu=e^{\int p\,dx};\qquad y'=ky\Rightarrow y=y_0e^{kt}f̂(ξ) = ∫ f(x) e^(−2πixξ) dx; Gaussian ↔ Gaussian; box ↔ sinc
A narrow function has a wide spectrum (uncertainty principle).
Worked example: xi = 0.5 → 0.455938 ✓ checked
See also: discrete Fourier transform · convolution theorem
\hat f(\xi)=\int f(x)e^{-2\pi ix\xi}dx;\quad e^{-\pi x^2}\leftrightarrow e^{-\pi\xi^2};\ \ \mathbf 1_{[-\frac12,\frac12]}\leftrightarrow\operatorname{sinc}(\xi)∫ₐᵇ f′ = f(b) − f(a)
Worked example: a = 0, b = 1 → 1 ✓ checked
See also: integration by parts
\int_a^bf'(x)\,dx=f(b)-f(a),\qquad \frac d{dx}\int_a^xf(t)\,dt=f(x)Γ(z) = ∫ t^(z−1) e^(−t) dt; Γ(z + 1) = zΓ(z); Γ(n) = (n − 1)!
The factorial for real numbers.
Worked example: z = 3.5 → 3.32335 ✓ checked
See also: beta function · Stirling's approximation
\Gamma(z)=\int_0^\infty t^{z-1}e^{-t}\,dt,\quad \Gamma(z+1)=z\,\Gamma(z),\quad \Gamma(n)=(n-1)!,\quad \Gamma(\tfrac12)=\sqrt\pi∫ e^(−ax²) dx = √(π/a); ∫ x² e^(−ax²) dx = √π/(2a^(3/2))
Worked example: a = 2 → [1.25331, 0.313329] ✓ checked
See also: Gaussian integral
\int_{-\infty}^{\infty}e^{-ax^2}dx=\sqrt{\frac\pi a},\quad \int_{-\infty}^{\infty}x^2e^{-ax^2}dx=\frac{\sqrt\pi}{2a^{3/2}}(1 + x)^α = Σ C(α, k) xᵏ
First-order: (1 + x)^α ≈ 1 + αx.
Worked example: x = 0.2, a = 0.5 → 1.09545 ✓ checked
See also: binomial theorem
(1+x)^\alpha=\sum_{k=0}^{\infty}\binom\alpha kx^k\quad(|x|<1),\qquad \binom\alpha k=\frac{\alpha(\alpha-1)\cdots(\alpha-k+1)}{k!}∇f, ∇·F, ∇×F
Laplacian: ∇²f = ∇·∇f.
Worked example: x = 1, y = 2 → 6 ✓ checked
See also: divergence theorem and Stokes' theorem
\nabla f=\Big(\frac{\partial f}{\partial x_i}\Big),\quad \nabla\!\cdot\mathbf F=\sum_i\frac{\partial F_i}{\partial x_i},\quad \nabla\times\mathbf F=\Big(\partial_yF_z-\partial_zF_y,\ \partial_zF_x-\partial_xF_z,\ \partial_xF_y-\partial_yF_x\Big)Hₙ = Σ 1/k ≈ ln n + γ + 1/(2n)
Diverges, but only logarithmically. The coupon-collector expectation is nHₙ.
Worked example: n = 1000 → 7.48547 ✓ checked
See also: coupon collector
H_n=\sum_{k=1}^n\frac1k\approx\ln n+\gamma+\frac1{2n},\quad \gamma\approx0.5772Hᵢⱼ = ∂²f/∂xᵢ∂xⱼ; ∇f = 0 and H ≻ 0 ⇒ local minimum
Eigenvalues of H give the curvature along principal directions; a mix of signs means a saddle.
Worked example: x = 1, y = 2 → [4, 2, 12] ✓ checked
See also: Newton's method (multivariate)
H_{ij}=\frac{\partial^2f}{\partial x_i\partial x_j};\quad \nabla f=0,\ H\succ0\Rightarrow\text{local minimum}dy/dx = −F_x/F_y
Worked example: x = 3, y = 4 → -0.75 ✓ checked
See also: chain rule
F(x,y)=0\ \Rightarrow\ \frac{dy}{dx}=-\frac{F_x}{F_y}∫ xⁿ dx = xⁿ⁺¹/(n + 1) + C; ∫ dx/x = ln|x| + C
Worked example: a = 1, b = 3, n = 2 → [8.66667, 1.09861] ✓ checked
See also: integrals of exp and trigonometric functions
\int x^n\,dx=\frac{x^{n+1}}{n+1}+C\ (n\ne-1),\qquad \int\frac{dx}{x}=\ln|x|+C∫ e^(ax) = e^(ax)/a; ∫ sin = −cos; ∫ cos = sin; ∫ 1/(1 + x²) = arctan x
Worked example: b = 1 → [3.19453, 0.459698, 0.841471, 0.785398] ✓ checked
See also: integral of a power
\int e^{ax}dx=\frac{e^{ax}}a,\ \int\sin x\,dx=-\cos x,\ \int\cos x\,dx=\sin x,\ \int\frac{dx}{1+x^2}=\arctan x∫ u dv = uv − ∫ v du
Example: ∫ x eˣ dx = (x − 1)eˣ + C.
Worked example: b = 1 → 1 ✓ checked
See also: substitution (change of variables)
\int u\,dv=uv-\int v\,duJᵢⱼ = ∂fᵢ/∂xⱼ
The best linear approximation of a vector function; backpropagation multiplies Jacobians.
Worked example: x = 1, y = 2 → [[2, 1], [1, 4]] ✓ checked
See also: multivariable chain rule
J_{ij}=\frac{\partial f_i}{\partial x_j}lim f/g = lim f′/g′
Worked example: x = 0.0001 → 0.5 ✓ checked
See also: classic limits
\lim_{x\to a}\frac{f(x)}{g(x)}=\lim_{x\to a}\frac{f'(x)}{g'(x)}\quad\text{(for }0/0\text{ or }\infty/\infty)p(x) = Σ yᵢ Π_{j≠i} (x − xⱼ)/(xᵢ − xⱼ)The unique degree-n polynomial through n + 1 points.
Worked example: xs = [0, 1, 2], ys = [1, 3, 7], x = 3 → 13 ✓ checked
See also: Newton divided differences
p(x)=\sum_{i}y_i\prod_{j\ne i}\frac{x-x_j}{x_i-x_j}L{1} = 1/s; L{eᵃᵗ} = 1/(s − a); L{tⁿ} = n!/sⁿ⁺¹; L{sin ωt} = ω/(s² + ω²)Worked example: s = 2 → [0.5, 0.25, 0.230769] ✓ checked
See also: first-order linear ODE
\mathcal L\{f\}(s)=\int_0^\infty f(t)e^{-st}dt;\ \ \mathcal L\{1\}=\frac1s,\ \mathcal L\{e^{at}\}=\frac1{s-a},\ \mathcal L\{t^n\}=\frac{n!}{s^{n+1}},\ \mathcal L\{\sin\omega t\}=\frac\omega{s^2+\omega^2}differentiate under the integral sign
Worked example: t = 1 → -0.26426 ✓ checked
See also: fundamental theorem of calculus
\frac d{dt}\int_{a(t)}^{b(t)}f(x,t)\,dx=f(b,t)b'-f(a,t)a'+\int_{a}^{b}\frac{\partial f}{\partial t}dxf′(c) = (f(b) − f(a))/(b − a)
Worked example: a = 0, b = 2 → 2 ✓ checked
See also: Taylor series
f'(c)=\frac{f(b)-f(a)}{b-a}\ \text{for some }c\in(a,b)‖f + g‖_p ≤ ‖f‖_p + ‖g‖_p
Worked example: x = [1, -2, 3], y = [2, 1, -1], p = 3 → 2.15443 ✓ checked
See also: Lp norm
\|f+g\|_p\le\|f\|_p+\|g\|_p\quad(p\ge1)∂z/∂t = Σ (∂z/∂xᵢ)(∂xᵢ/∂t); J_{f∘g} = J_f J_gWorked example: t = 1 → 10.3781 ✓ checked
See also: Jacobian matrix · chain rule
\frac{\partial z}{\partial t}=\sum_i\frac{\partial z}{\partial x_i}\frac{\partial x_i}{\partial t},\qquad J_{f\circ g}=J_f(g)\,J_gpiecewise cubics with continuous value, slope and curvature
Worked example: h = 1 → 1 ✓ checked
See also: Lagrange interpolation
S\in C^2,\ \ S_i(x_i)=y_i,\ \ S''(x_0)=S''(x_n)=0p(x) = f[x₀] + f[x₀,x₁](x − x₀) + …
Worked example: x = 3 → 13 ✓ checked
See also: Lagrange interpolation
p(x)=f[x_0]+f[x_0,x_1](x-x_0)+f[x_0,x_1,x_2](x-x_0)(x-x_1)+\cdots,\quad f[x_i,x_j]=\frac{f_j-f_i}{x_j-x_i}x ← x − H⁻¹ ∇f
Quadratic convergence near a minimiser; one step solves a quadratic exactly.
Worked example: H = [[2, 1], [1, 3]], b = [1, 2], x = [5, -3] → [0.2, 0.6] ✓ checked
See also: Newton's method (root finding) · Hessian matrix and second-derivative test
\mathbf x\leftarrow\mathbf x-H^{-1}\nabla fxₙ₊₁ = xₙ − f(xₙ)/f′(xₙ)
For minimisation apply it to f′: x ← x − f′/f″. Converges quadratically near a simple root.
Worked example: x = 1 → 1.5 ✓ checked
See also: Taylor series
x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}d/dx xⁿ = n xⁿ⁻¹
Holds for any real n.
Worked example: x = 2, n = 3 → 12 ✓ checked
See also: product rule
\frac{d}{dx}x^n=n\,x^{n-1}(fg)′ = f′g + fg′
Worked example: x = 1 → 2.22324 ✓ checked
See also: quotient rule · chain rule
(fg)'=f'g+fg'(f/g)′ = (f′g − fg′)/g²
Worked example: x = 1 → -0.150584 ✓ checked
See also: product rule
\Big(\frac fg\Big)'=\frac{f'g-fg'}{g^2}L = lim |aₙ₊₁/aₙ| : L < 1 converges, L > 1 diverges
Worked example: n = 20 → 0.047619 ✓ checked
See also: geometric series
L=\lim\Big|\frac{a_{n+1}}{a_n}\Big|\ \text{or}\ \lim|a_n|^{1/n}:\ L<1\Rightarrow\text{converges},\ L>1\Rightarrow\text{diverges}D ≈ (4D(h/2) − D(h))/3
Worked example: x = 1, h = 0.1 → 2.71828 ✓ checked
See also: finite differences
D\approx\frac{4D(h/2)-D(h)}{3}\quad(\text{cancels the }h^2\text{ term})characteristic roots r = (−b ± √(b² − 4ac))/2a
Complex roots α ± iβ give e^(αx)(C₁ cos βx + C₂ sin βx): damped oscillation.
Worked example: a = 1, b = 0, c = 4, x = 1 → -0.416151 ✓ checked
See also: first-order linear ODE
ay''+by'+cy=0:\ r=\frac{-b\pm\sqrt{b^2-4ac}}{2a};\ \ y=C_1e^{r_1x}+C_2e^{r_2x}eˣ, sin, cos, ln(1 + x), 1/(1 − x), (1 + x)^α
ln(1 + x) and 1/(1 − x) need |x| < 1; the others hold for all x.
Worked example: x = 0.3 → [1.34986, 0.29552, 0.955336, 0.262364, 1.42857] ✓ checked
See also: Taylor series
e^x=\sum\frac{x^n}{n!},\ \sin x=\sum\frac{(-1)^nx^{2n+1}}{(2n+1)!},\ \cos x=\sum\frac{(-1)^nx^{2n}}{(2n)!},\ \ln(1+x)=\sum\frac{(-1)^{n+1}x^n}{n},\ \frac1{1-x}=\sum x^n∫ f(g(x)) g′(x) dx = ∫ f(u) du
Worked example: b = 1 → 0.841471 ✓ checked
See also: change of variables (Jacobian)
\int_a^bf(g(x))\,g'(x)\,dx=\int_{g(a)}^{g(b)}f(u)\,duΣk = n(n+1)/2; Σk² = n(n+1)(2n+1)/6; Σk³ = (n(n+1)/2)²
Worked example: n = 10 → [55, 385, 3025] ✓ checked
See also: arithmetic series
\sum_{k=1}^nk=\frac{n(n+1)}2,\quad \sum k^2=\frac{n(n+1)(2n+1)}6,\quad \sum k^3=\Big(\frac{n(n+1)}2\Big)^2Rₙ(x) = f⁽ⁿ⁺¹⁾(ξ)/(n + 1)! · (x − a)ⁿ⁺¹
Bounds the error of a Taylor polynomial.
Worked example: x = 0.5, n = 3 → 0.00288794 ✓ checked
See also: Taylor series
R_n(x)=\frac{f^{(n+1)}(\xi)}{(n+1)!}(x-a)^{n+1}\ \text{for some }\xi\text{ between }a,xf(x) ≈ Σₙ f⁽ⁿ⁾(a) (x−a)ⁿ / n!
where expansion about a
Remainder is O((x−a)^(N+1)).
Worked example: x = 0.5 → 1.64844 ✓ checked
See also: chain rule · Newton's method (root finding)
f(x)\approx\sum_{n=0}^{N}\frac{f^{(n)}(a)}{n!}(x-a)^ntrapezoid: h/2 [f₀ + 2Σfᵢ + fₙ]; Simpson: h/3 [f₀ + 4f₁ + 2f₂ + … + fₙ]
Errors O(h²) and O(h⁴).
Worked example: n = 10 → 1.71828 ✓ checked
See also: Euler–Maclaurin formula
\int_a^bf\approx\frac h2\Big[f(a)+2\sum_{i=1}^{n-1}f(x_i)+f(b)\Big],\qquad \int_a^bf\approx\frac h3\Big[f_0+4f_1+2f_2+\cdots+4f_{n-1}+f_n\Big]|a + b| ≤ |a| + |b|
Worked example: a = -3, b = 5 → 6 ✓ checked
See also: Minkowski inequality (triangle inequality in Lp)
|a+b|\le|a|+|b|,\qquad \big||a|-|b|\big|\le|a-b|1 + x ≤ eˣ; ln x ≤ x − 1
Used constantly in proofs of concentration bounds.
Worked example: x = 0.5 → [0.148721, 0.193147] ✓ checked
See also: Jensen's inequality
1+x\le e^x,\qquad \ln x\le x-1,\qquad e^x\ge1+x+\tfrac{x^2}2\ (x\ge0),\qquad 1-x\le e^{-x}π/2 = Π 4n²/(4n² − 1)
Worked example: N = 100000 → 1.57079 ✓ checked
See also: Basel problem and related sums
\frac\pi2=\prod_{n=1}^{\infty}\frac{4n^2}{4n^2-1}ab ≤ aᵖ/p + bᵠ/q
p = q = 2 gives ab ≤ (a² + b²)/2.
Worked example: a = 2, b = 3, p = 3, q = 1.5 → 0.130768 ✓ checked
See also: Cauchy–Schwarz and Hölder inequalities
ab\le\frac{a^p}p+\frac{b^q}q\quad\Big(\frac1p+\frac1q=1\Big)S = 1/((1 − p) + p/s)
where p = parallelisable fraction, s = speed-up of that part
The serial fraction caps the speed-up at 1/(1 − p).
Worked example: p = 0.9, s = 10 → 5.26316 ✓ checked
See also: Gustafson's law
S=\frac1{(1-p)+p/s}average f(U) and f(1 − U)
Reduces variance when f is monotone, since the pair is negatively correlated.
Worked example: u = 0.2 → 1.72347 ✓ checked
See also: control variates
\hat\theta=\frac1{2N}\sum_i\big[f(U_i)+f(1-U_i)\big]h*(x) = argmax P(y = k | x); R* = 1 − E max P(y|x)
No classifier can beat it.
Worked example: p = [0.9, 0.6, 0.8] → 0.233333 ✓ checked
See also: Bayes' theorem
h^*(x)=\arg\max_kP(y=k\mid x),\qquad R^*=1-\mathbb E_x\Big[\max_kP(y=k\mid x)\Big]Z₀ = √(−2 ln U₁) cos(2πU₂), Z₁ = √(−2 ln U₁) sin(2πU₂)
where U₁, U₂ ~ Uniform(0, 1)
Two independent standard normals from two uniforms.
Worked example: u1 = 0.5, u2 = 0 → 1.17741 ✓ checked
See also: inverse-transform sampling
Z_0=\sqrt{-2\ln U_1}\,\cos(2\pi U_2),\qquad Z_1=\sqrt{-2\ln U_1}\,\sin(2\pi U_2)1 − cos x = 2 sin²(x/2); use log1p, expm1
Subtracting nearly equal numbers wipes out significant digits.
Worked example: x = 1e-08 → [0, 5e-17, 1e-08] ✓ checked
See also: machine epsilon and floating-point precision
1-\cos x=2\sin^2\!\frac x2,\qquad \ln(1+x)\to\operatorname{log1p}(x),\qquad e^x-1\to\operatorname{expm1}(x)κ = |x f′(x)/f(x)|
How much a relative input error is amplified. √x has κ = ½; subtracting nearby numbers has huge κ.
Worked example: x = 4 → 0.5 ✓ checked
See also: condition number
\kappa_f(x)=\Big|\frac{x\,f'(x)}{f(x)}\Big|,\qquad \frac{\delta f}{f}\approx\kappa_f\frac{\delta x}xvariance shrinks by a factor (1 − ρ²)
Worked example: rho = 0.9 → 0.19 ✓ checked
See also: antithetic variates
\hat\theta=\bar Y-c\big(\bar Z-\mathbb E Z\big),\ \ c^*=\frac{\operatorname{Cov}(Y,Z)}{\operatorname{Var}Z},\ \ \operatorname{Var}\hat\theta=(1-\rho^2)\operatorname{Var}\bar Yd = 1 − cos θ
Not a metric (no triangle inequality); angular distance is.
Worked example: x = [1, 0], y = [1, 1] → 0.292893 ✓ checked
See also: cosine similarity
d_{\cos}=1-\cos\theta,\qquad d_{\text{ang}}=\frac{\arccos(\cos\theta)}\piR* ≤ R_1NN ≤ 2R*(1 − R*)
where R* = Bayes error
With enough data, nearest-neighbour error is at most twice the best possible.
Worked example: r = 0.1 → 0.18 ✓ checked
See also: Bayes error and optimal classifier
R^*\le R_{\mathrm{1NN}}\le2R^*\,(1-R^*)ESS = (Σwᵢ)²/Σwᵢ²
N when the weights are equal, 1 when one weight dominates.
Worked example: w = [1, 1, 1, 1] → 4 ✓ checked
See also: importance sampling
\mathrm{ESS}=\frac{\big(\sum_iw_i\big)^2}{\sum_iw_i^2}DFT O(N²) → FFT O(N log₂ N)
Worked example: N = 1024 → [1.04858e+06, 10240] ✓ checked
See also: discrete Fourier transform
\text{DFT: }O(N^2)\ \ \longrightarrow\ \ \text{FFT: }O(N\log_2N)x = (−1)ˢ (1.m)₂ 2^(e − bias)
bfloat16 keeps float32's range with less precision, which is why it needs no loss scaling.
Worked example → [65504, 3.40282e+38] ✓ checked
See also: machine epsilon and floating-point precision
x=(-1)^s\,(1.m)_2\,2^{e-\text{bias}};\quad \text{float16: 1+5+10 bits, max }65504;\ \ \text{bfloat16: 1+8+7 bits;}\ \ \text{float32: 1+8+23}R̂ = √((n − 1)/n + B/(nW))
where B = between-chain variance, W = within-chain variance, n = chain length
Values close to 1 (below about 1.01) indicate convergence.
Worked example: n = 1000, B = 0.5, W = 1 → 0.99975 ✓ checked
See also: MCMC effective sample size
\hat R=\sqrt{\frac{n-1}n+\frac{B}{nW}}x | y ~ N(ρy, 1 − ρ²)
where unit variances, correlation ρ
Mixing slows as |ρ| → 1.
Worked example: rho = 0.8, y = 1 → [0.8, 0.36] ✓ checked
See also: Metropolis–Hastings acceptance · multivariate normal: conditional distribution
x\mid y\sim\mathcal N\big(\rho y,\ 1-\rho^2\big),\qquad y\mid x\sim\mathcal N\big(\rho x,\ 1-\rho^2\big)S = (1 − p) + pN
where N = processors, p = parallel fraction of the scaled workload
Worked example: p = 0.9, N = 100 → 90.1 ✓ checked
See also: Amdahl's law
S=(1-p)+pNhalf-step momentum, full-step position, half-step momentum
where U = −log π
Reversible and volume-preserving; the accept step corrects the small energy error.
Worked example: q = 1, p = 0, e = 0.1 → [0.995, -0.09975] ✓ checked
See also: Metropolis–Hastings acceptance · Langevin dynamics
p\leftarrow p-\tfrac\varepsilon2\nabla U(q),\quad q\leftarrow q+\varepsilon p,\quad p\leftarrow p-\tfrac\varepsilon2\nabla U(q)d_H = number of positions where a and b differ
Worked example: a = 'karolin', b = 'kathrin' → 3 ✓ checked
See also: Levenshtein distance recurrence
d_H(a,b)=\sum_i\mathbf 1[a_i\ne b_i]k = ⌈log₂ n⌉ + 1; h = 2 IQR n^(−1/3)
Worked example: n = 1000, iqr = 2 → [11, 0.4] ✓ checked
See also: kernel density estimate and Silverman's rule
\text{Sturges: }k=\lceil\log_2n\rceil+1,\qquad \text{Freedman–Diaconis: }h=2\,\mathrm{IQR}\,n^{-1/3}E_p[f] ≈ (1/N) Σ f(xᵢ) p(xᵢ)/q(xᵢ)
where q must cover the support of p·f
The self-normalised form divides by Σw. Off-policy RL uses the same trick.
Worked example: x = [0.5, 1.5, 2.5], w = [0.5, 1, 2] → 2.25 ✓ checked
See also: effective sample size · Horvitz–Thompson estimator
\mathbb E_p[f]=\mathbb E_q\Big[f(x)\frac{p(x)}{q(x)}\Big]\approx\frac1N\sum_if(x_i)\,w_i,\quad w_i=\frac{p(x_i)}{q(x_i)}X = F⁻¹(U)
Worked example: u = 0.5, lam = 2 → 0.346574 ✓ checked
See also: Box–Muller transform · exponential distribution
U\sim\mathrm U(0,1)\ \Rightarrow\ X=F^{-1}(U)\sim F;\qquad \text{exponential: }X=-\frac{\ln(1-U)}\lambday = xᵢ − c; t = s + y; c = (t − s) − y; s = t
Reduces the rounding error of a long sum from O(nε) to O(ε).
Worked example: n = 10 → 1 ✓ checked
See also: machine epsilon and floating-point precision
y=x_i-c,\quad t=s+y,\quad c=(t-s)-y,\quad s=tf̂(x) = (1/nh) Σ K((x − xᵢ)/h)
where K = kernel, e.g. Gaussian
Worked example: x = [0, 1], pt = 0.5, h = 0.5 → 0.483941 ✓ checked
See also: histogram bin width rules
\hat f(x)=\frac1{nh}\sum_iK\Big(\frac{x-x_i}h\Big),\qquad h=0.9\min\Big(\hat\sigma,\frac{\mathrm{IQR}}{1.34}\Big)n^{-1/5}x ← x − (ε/2)∇U(x) + √ε ξ
Samples from exp(−U); the score-based view of diffusion models uses ∇log p in place of −∇U.
Worked example: x = 1, e = 0.01, noise = 0 → 0.995 ✓ checked
See also: Hamiltonian Monte Carlo leapfrog step · denoising score matching
x_{k+1}=x_k-\tfrac\varepsilon2\nabla U(x_k)+\sqrt\varepsilon\,\xi_k,\quad \xi_k\sim\mathcal N(0,I)ε = 2^(−52) (float64), 2^(−23) (float32), 2^(−10) (float16)
The gap between 1 and the next representable number. Relative rounding error is at most ε/2.
Worked example → [2.22045e-16, 1.19209e-07, 0.000976562] ✓ checked
See also: catastrophic cancellation and stable rewrites
\epsilon_{\text{mach}}=2^{-p+1}:\ \ \text{float64 }2^{-52}\approx2.2\times10^{-16},\ \ \text{float32 }2^{-23}\approx1.2\times10^{-7},\ \ \text{float16 }2^{-10}\approx9.8\times10^{-4}FLOPs = 2·m·n·p
A multiply and an add per term. Strassen: O(n^{log₂7}) ≈ O(n^{2.807}).
Worked example: m = 1024, n = 4096, p = 4096 → 3.43597e+10 ✓ checked
See also: transformer parameter count
\text{FLOPs}(A_{m\times n}B_{n\times p})=2mnpESS = N/(1 + 2 Σ ρₖ)
where ρₖ = autocorrelation at lag k
Worked example: N = 10000, rho = [0.5, 0.25, 0.125] → 3636.36 ✓ checked
See also: autocorrelation function · effective sample size
\mathrm{ESS}=\frac N{1+2\sum_{k\ge1}\rho_k}α = min(1, π(x′)q(x|x′)/(π(x)q(x′|x)))
where π needs to be known only up to a constant
A symmetric proposal reduces it to min(1, π(x′)/π(x)).
Worked example: px = 0.2, pn = 0.1 → 0.5 ✓ checked
See also: Gibbs sampling (bivariate normal) · Hamiltonian Monte Carlo leapfrog step
\alpha=\min\Big(1,\ \frac{\pi(x')\,q(x\mid x')}{\pi(x)\,q(x'\mid x)}\Big)min–max to [0, 1]; z-score; robust scaling
Fit the statistics on the training set only.
Worked example: x = [2, 4, 6, 10] → [0.5, 0.169031] ✓ checked
See also: z-score · interquartile range
x'=\frac{x-x_{\min}}{x_{\max}-x_{\min}},\qquad z=\frac{x-\mu}\sigma,\qquad x'=\frac{x-\operatorname{median}}{\mathrm{IQR}}SE = σ/√N
Error falls as N^(−½) whatever the dimension: ten times the accuracy costs a hundred times the samples.
Worked example: s = 2, N = 10000 → 0.02 ✓ checked
See also: standard error of the mean
\mathrm{SE}=\frac\sigma{\sqrt N}ŷ(x) = Σ K(x − xᵢ)yᵢ / Σ K(x − xᵢ)
A locally weighted average; attention is its soft, learned cousin.
Worked example: x = [0, 1], y = [0, 2], pt = 0.25, h = 0.5 → 0.537883 ✓ checked
See also: scaled dot-product attention · kernel density estimate and Silverman's rule
\hat y(x)=\frac{\sum_iK_h(x-x_i)\,y_i}{\sum_iK_h(x-x_i)}accept if u ≤ f(x)/(M g(x)); acceptance rate = 1/M
where f ≤ M g everywhere
Worked example: M = 2.5 → 0.4 ✓ checked
See also: importance sampling
\text{draw }x\sim g,\ u\sim\mathrm U(0,1);\ \text{accept if }u\le\frac{f(x)}{M\,g(x)};\qquad \text{acceptance rate}=\frac1MP(item in sample) = k/n
Keep item i with probability k/i, replacing a random slot: uniform sample of a stream of unknown length.
Worked example: k = 10, n = 1000 → 0.01 ✓ checked
See also: Monte Carlo standard error
P(\text{item }i\text{ is in the sample of size }k)=\frac knP = min(P_peak, β·I)
where β = memory bandwidth, I = arithmetic intensity
Below the ridge point a kernel is memory-bound; above it compute-bound.
Worked example: peak = 1e+14, bw = 2e+12, I = 25 → 5e+13 ✓ checked
See also: matrix-multiplication cost
P=\min\big(P_{\text{peak}},\ \beta\cdot I\big),\qquad I=\frac{\text{FLOPs}}{\text{bytes moved}}average of the last w values
An FIR low-pass filter. Exponential smoothing weights recent values more.
Worked example: x = [1, 2, 3, 4, 5], w = 3 → 4 ✓ checked
See also: simple exponential smoothing · FIR and IIR filters
\mathrm{SMA}_t=\frac1w\sum_{i=0}^{w-1}x_{t-i},\qquad \mathrm{WMA}_t=\frac{\sum_iw_ix_{t-i}}{\sum_iw_i}subtract max(z) before exponentiating
Without it, exp(1000) overflows to infinity.
Worked example: z = [1000, 999] → [0.731059, 0.268941] ✓ checked
See also: log-sum-exp · softmax
\operatorname{softmax}(z)_i=\frac{e^{z_i-m}}{\sum_je^{z_j-m}},\qquad \operatorname{LSE}(z)=m+\ln\sum_je^{z_j-m},\quad m=\max_jz_jenvelope = |x + jH{x}|Worked example: t = 0.3 → 1 ✓ checked
See also: discrete Fourier transform
x_a(t)=x(t)+j\mathcal H\{x\}(t),\qquad \text{envelope}=|x_a(t)|,\quad \text{phase}=\arg x_a2f_H/n ≤ fs ≤ 2f_L/(n − 1)
Worked example: fh = 25, fl = 20, n = 4 → [12.5, 13.3333] ✓ checked
See also: Nyquist rate and aliasing
\frac{2f_H}{n}\le f_s\le\frac{2f_L}{n-1},\quad 1\le n\le\Big\lfloor\frac{f_H}{B}\Big\rfloors = (2/T)(z − 1)/(z + 1)
Maps the analogue left half-plane into the unit disc, preserving stability.
Worked example: T = 0.1, z = 0.9 → -1.05263 ✓ checked
See also: z-transform and stability
s=\frac2T\,\frac{z-1}{z+1}|H|² = 1/(1 + (ω/ωc)^(2n))
Maximally flat in the pass band; −3 dB at ωc.
Worked example: w = 2, wc = 1, n = 2 → 0.0588235 ✓ checked
See also: first-order low-pass filter
|H(j\omega)|^2=\frac1{1+(\omega/\omega_c)^{2n}}DFT(f ∗ g) = DFT(f) · DFT(g) (circular convolution)
FFT convolution costs O(N log N) instead of O(N²).
Worked example: f = [1, 2, 3, 0], g = [0, 1, 0.5, 0] → 0 ✓ checked
See also: discrete convolution · discrete Fourier transform
\mathcal F\{f*g\}=\mathcal F\{f\}\cdot\mathcal F\{g\}r_xy[k] = Σ x[n] y[n + k]
The peak lag gives the delay between two signals. Wiener–Khinchin: the PSD is the Fourier transform of the autocorrelation.
Worked example: x = [1, 2, 3], y = [0, 1, 2], k = 1 → 5 ✓ checked
See also: autocorrelation function · matched filter
r_{xy}[k]=\sum_nx[n]\,y[n+k]Δf = fs/N; fₖ = kΔf
Zero-padding interpolates the spectrum but does not add resolution.
Worked example: fs = 44100, N = 4096, k = 100 → [10.7666, 1076.66] ✓ checked
See also: short-time Fourier transform
\Delta f=\frac{f_s}N,\qquad f_k=k\,\Delta fy[n] = Σ bₖ x[n − k] (− Σ aₖ y[n − k] for IIR)
FIR filters are always stable and can have linear phase.
Worked example: b = [0.5, 0.5], x = [1, 3, 5] → [0.5, 2, 4] ✓ checked
See also: discrete convolution · z-transform and stability
y[n]=\sum_{k=0}^{M}b_kx[n-k]\ \ (\text{FIR}),\qquad y[n]=\sum_kb_kx[n-k]-\sum_{k\ge1}a_ky[n-k]\ \ (\text{IIR})fc = 1/(2πRC); yₙ = yₙ₋₁ + α(xₙ − yₙ₋₁)
The digital version is an exponential moving average.
Worked example: R = 1000, C = 1e-06 → 159.155 ✓ checked
See also: simple exponential smoothing
f_c=\frac1{2\pi RC},\qquad y_n=y_{n-1}+\alpha(x_n-y_{n-1}),\ \ \alpha=\frac{\Delta t}{RC+\Delta t}square wave = (4/π) Σ sin(kωt)/k over odd k
The partial sums overshoot by about 8.9% at the jump (Gibbs phenomenon).
Worked example: N = 200 → 1.17899 ✓ checked
See also: discrete Fourier transform
x(t)=\frac4\pi\sum_{k\ \text{odd}}\frac{\sin(k\omega t)}{k},\qquad c_n=\frac1T\int_0^Tx(t)e^{-jn\omega t}dta = (x₁ + x₂)/√2; d = (x₁ − x₂)/√2
Orthonormal: energy is preserved. Recurse on the averages for a multi-level decomposition.
Worked example: x1 = 3, x2 = 1 → [2.82843, 1.41421] ✓ checked
See also: Parseval's theorem (DFT)
a=\frac{x_1+x_2}{\sqrt2},\qquad d=\frac{x_1-x_2}{\sqrt2}f(t) = f₀ + kt; phase = 2π(f₀t + ½kt²)
Worked example: f0 = 10, k = 4, t = 2 → [18, 175.929] ✓ checked
See also: short-time Fourier transform
f(t)=f_0+kt,\qquad \phi(t)=2\pi\Big(f_0t+\tfrac12kt^2\Big)h(t) = s(T − t); SNR_max = 2E/N₀
Maximises the output SNR for a known signal in white noise.
Worked example: E = 4, N0 = 0.5 → 16 ✓ checked
See also: cross-correlation
h(t)=s(T-t),\qquad \mathrm{SNR}_{\max}=\frac{2E_s}{N_0}m = 2595 log₁₀(1 + f/700)
Approximates how pitch is perceived; 1000 Hz ≈ 1000 mel.
Worked example: f = 1000 → 999.986 ✓ checked
See also: mel-frequency cepstral coefficients
m=2595\log_{10}\Big(1+\frac f{700}\Big),\qquad f=700\big(10^{m/2595}-1\big)DCT-II of the log mel-band energies
Worked example: logE = [1, 2, 3], k = 1 → -1.73205 ✓ checked
See also: mel scale
c_k=\sum_{m=1}^{M}\ln E_m\,\cos\!\Big[\frac{\pi k}{M}\big(m-\tfrac12\big)\Big]fs ≥ 2 f_max; f_alias = |f − fs·round(f/fs)|
Sampling below the Nyquist rate folds high frequencies onto low ones.
Worked example: f = 70, fs = 100 → 30 ✓ checked
See also: discrete Fourier transform · Shannon–Hartley capacity (AWGN)
f_s\ge2f_{\max},\qquad f_{\text{alias}}=\big|f-f_s\operatorname{round}(f/f_s)\big|Σ|xₙ|² = (1/N) Σ|Xₖ|²
Energy is the same in the time and frequency domains.
Worked example: x = [1, 2, 3, 4] → [30, 30] ✓ checked
See also: Parseval's identity for an orthonormal basis
\sum_{n=0}^{N-1}|x_n|^2=\frac1N\sum_{k=0}^{N-1}|X_k|^2Ŝ = |X_k|²/(N·fs)
Welch's method averages periodograms of windowed segments to cut the variance.
Worked example: X = 8, N = 64, fs = 1 → 1 ✓ checked
See also: short-time Fourier transform
\hat S_{xx}(f_k)=\frac1{N f_s}\,|X_k|^2SQNR ≈ 6.02b + 1.76 dB
where b = bits
Each extra bit adds about 6 dB.
Worked example: b = 16 → 98.08 ✓ checked
See also: signal-to-noise ratio in decibels · affine quantisation
\mathrm{SQNR}\approx6.02\,b+1.76\ \text{dB}windowed DFT slid along the signal with hop H
Frequency resolution Δf = fs/N, time resolution about N/fs: they trade off.
Worked example: fs = 16000, N = 512 → [31.25, 32] ✓ checked
See also: window functions · time–frequency uncertainty principle
X(m,k)=\sum_{n}x[n]\,w[n-mH]\,e^{-2\pi ikn/N}Δt·Δf ≥ 1/(4π)
Equality for a Gaussian pulse: you cannot localise a signal in both time and frequency.
Worked example: dt = 0.01 → 7.95775 ✓ checked
See also: Fourier transform pairs
\Delta t\,\Delta f\ge\frac1{4\pi}x(t) = Σ x[n] sinc((t − nT)/T)
Perfect reconstruction of a band-limited signal from its samples.
Worked example: x = [1, 0, 0], t = 0 → 1 ✓ checked
See also: Nyquist rate and aliasing
x(t)=\sum_nx[n]\,\operatorname{sinc}\!\Big(\frac{t-nT}T\Big)Hann: ½(1 − cos(2πn/(N − 1))); Hamming: 0.54 − 0.46 cos(2πn/(N − 1))
Tapering the edges reduces spectral leakage.
Worked example: n = 4, N = 9 → [1, 1] ✓ checked
See also: short-time Fourier transform
w_{\text{Hann}}[n]=\tfrac12\Big(1-\cos\frac{2\pi n}{N-1}\Big),\quad w_{\text{Hamming}}[n]=0.54-0.46\cos\frac{2\pi n}{N-1}causal LTI system is stable iff its poles lie inside the unit circle
Worked example: a1 = -0.9 → 0.9 ✓ checked
See also: FIR and IIR filters
X(z)=\sum_nx[n]z^{-n},\qquad H(z)=\frac{\sum b_kz^{-k}}{1+\sum a_kz^{-k}},\ \ \text{stable iff all poles satisfy }|z|<1sin(a ± b), cos(a ± b), tan(a + b)
Worked example: a = 0.4, b = 0.9 → [-1.11022e-16, 5.55112e-17, 2.22045e-15] ✓ checked
See also: double- and half-angle formulas
\sin(a\pm b)=\sin a\cos b\pm\cos a\sin b,\quad \cos(a\pm b)=\cos a\cos b\mp\sin a\sin b,\quad \tan(a+b)=\frac{\tan a+\tan b}{1-\tan a\tan b}s = rθ; sector area = ½r²θ
Degrees to radians: θ = d·π/180.
Worked example: r = 2, th = 1.5708 → [3.14159, 3.14159] ✓ checked
See also: circle and sphere
s=r\theta,\qquad A_{\text{sector}}=\tfrac12r^2\theta\qquad(\theta\text{ in radians})circle: C = 2πr, A = πr²; sphere: S = 4πr², V = (4/3)πr³
Worked example: r = 2 → [12.5664, 12.5664, 50.2655, 33.5103] ✓ checked
See also: volume and surface of the n-ball
C=2\pi r,\ \ A=\pi r^2;\qquad S=4\pi r^2,\ \ V=\tfrac43\pi r^3|z| = √(a² + b²); multiply moduli, add arguments
Worked example: a = 3, b = 4 → [5, 0.927295] ✓ checked
See also: De Moivre's theorem and roots of unity · Euler's formula and identity
z=a+bi=re^{i\theta},\ \ r=\sqrt{a^2+b^2},\ \theta=\operatorname{atan2}(b,a),\quad z_1z_2=r_1r_2e^{i(\theta_1+\theta_2)},\quad \bar z=re^{-i\theta}cylinder πr²h; cone ⅓πr²h; pyramid ⅓A_b h
Worked example: r = 3, h = 4 → [113.097, 37.6991, 47.1239] ✓ checked
See also: circle and sphere
V_{\text{cyl}}=\pi r^2h,\quad V_{\text{cone}}=\tfrac13\pi r^2h,\quad V_{\text{pyr}}=\tfrac13A_bh,\quad A_{\text{cone side}}=\pi r\sqrt{r^2+h^2}Euclidean distance, midpoint, point-to-line distance
Worked example: x = [0, 0], y = [3, 4], abc = [3, 4, -10], p = [0, 0] → [5, 2] ✓ checked
See also: Euclidean (L2) norm
d=\sqrt{(x_2-x_1)^2+(y_2-y_1)^2},\quad M=\Big(\frac{x_1+x_2}2,\frac{y_1+y_2}2\Big),\quad d_{\text{pt-line}}=\frac{|ax_0+by_0+c|}{\sqrt{a^2+b^2}}sin 2θ = 2 sin θ cos θ; cos 2θ = cos²θ − sin²θ; sin²(θ/2) = (1 − cos θ)/2
Worked example: t = 0.7 → [-1.11022e-16, -2.77556e-17, 2.77556e-17] ✓ checked
See also: Pythagorean trigonometric identities
\sin2\theta=2\sin\theta\cos\theta,\quad \cos2\theta=\cos^2\theta-\sin^2\theta=1-2\sin^2\theta,\quad \sin^2\tfrac\theta2=\frac{1-\cos\theta}2area πab; perimeter ≈ π[3(a + b) − √((3a + b)(a + 3b))] (Ramanujan)
Worked example: a = 5, b = 3 → [47.1239, 25.527, 0.8] ✓ checked
See also: circle and sphere
A=\pi ab,\qquad P\approx\pi\Big[3(a+b)-\sqrt{(3a+b)(a+3b)}\Big],\qquad e=\sqrt{1-\frac{b^2}{a^2}}great-circle distance between two latitude/longitude points
where R = 6371 km
Worked example: lat1 = 0, lon1 = 0, lat2 = 0, lon2 = 90, R = 6371 → 10007.5 ✓ checked
See also: arc length and sector area
d=2R\arcsin\sqrt{\sin^2\!\frac{\Delta\varphi}2+\cos\varphi_1\cos\varphi_2\sin^2\!\frac{\Delta\lambda}2}A = √(s(s − a)(s − b)(s − c))
Worked example: a = 3, b = 4, c = 5 → 6 ✓ checked
See also: law of sines and cosines
A=\sqrt{s(s-a)(s-b)(s-c)},\quad s=\tfrac{a+b+c}2;\qquad A=\tfrac12ab\sin Ca/sin A = b/sin B = c/sin C; c² = a² + b² − 2ab cos C
where R = circumradius
Worked example: a = 3, b = 4, C = 1.5708 → 5 ✓ checked
See also: Heron's formula
\frac a{\sin A}=\frac b{\sin B}=\frac c{\sin C}=2R,\qquad c^2=a^2+b^2-2ab\cos Clog(ab) = log a + log b; log aⁿ = n log a
Worked example: a = 6, b = 4, n = 3 → [4.44089e-16, 0] ✓ checked
See also: change of base for logarithms
\log(ab)=\log a+\log b,\quad \log\frac ab=\log a-\log b,\quad \log a^n=n\log a,\quad a^{m+n}=a^ma^n,\quad (a^m)^n=a^{mn}d = |ax₀ + by₀ + cz₀ + d|/√(a² + b² + c²)
The same formula gives the margin of a linear classifier.
Worked example: n = [1, 2, 2], d = -3, p = [0, 0, 0] → 1 ✓ checked
See also: distance and midpoint · SVM margin width
d=\frac{|ax_0+by_0+cz_0+d|}{\sqrt{a^2+b^2+c^2}},\qquad \mathbf n\cdot(\mathbf x-\mathbf x_0)=0sin a cos b = ½[sin(a + b) + sin(a − b)]
Worked example: a = 0.4, b = 0.9 → [0, 0] ✓ checked
See also: angle addition formulas
\sin a\cos b=\tfrac12[\sin(a+b)+\sin(a-b)],\qquad \cos a+\cos b=2\cos\frac{a+b}2\cos\frac{a-b}2sin²θ + cos²θ = 1; 1 + tan²θ = sec²θ
Worked example: t = 0.7 → [1, 0] ✓ checked
See also: double- and half-angle formulas
\sin^2\theta+\cos^2\theta=1,\qquad 1+\tan^2\theta=\sec^2\theta,\qquad 1+\cot^2\theta=\csc^2\thetaΔ = b² − 4ac; vertex at x = −b/2a; sum of roots −b/a, product c/a
Worked example: a = 1, b = -3, c = 2 → [1, 1.5, 3, 2] ✓ checked
See also: quadratic formula
\Delta=b^2-4ac,\qquad \text{vertex}\ \Big(-\frac b{2a},\ c-\frac{b^2}{4a}\Big),\qquad x_1+x_2=-\frac ba,\ x_1x_2=\frac caA = (n/4)s² cot(π/n); diagonals = n(n − 3)/2
Worked example: n = 6, s = 2 → [10.3923, 120, 9] ✓ checked
See also: shoelace formula (polygon area)
A=\frac n4s^2\cot\frac\pi n,\qquad \text{interior angle}=\frac{(n-2)\pi}n,\qquad \text{diagonals}=\frac{n(n-3)}2A = ½|Σ (xᵢyᵢ₊₁ − xᵢ₊₁yᵢ)|
Worked example: x = [0, 4, 4, 0], y = [0, 0, 3, 3] → 12 ✓ checked
See also: Heron's formula
A=\tfrac12\Big|\sum_{i=1}^{n}\big(x_iy_{i+1}-x_{i+1}y_i\big)\Big|relations between the roots and the coefficients of ax³ + bx² + cx + d
Worked example: a = 1, b = -6, c = 11, d = -6 → [6, 11, 6] ✓ checked
See also: quadratic discriminant and vertex
x_1+x_2+x_3=-\frac ba,\quad x_1x_2+x_1x_3+x_2x_3=\frac ca,\quad x_1x_2x_3=-\frac daVₙ = π^(n/2) rⁿ/Γ(n/2 + 1)
In high dimensions nearly all of the volume sits near the surface, and the unit ball's volume tends to 0.
Worked example: n = 3, r = 1 → 4.18879 ✓ checked
See also: circle and sphere · gamma function
V_n(r)=\frac{\pi^{n/2}}{\Gamma\big(\frac n2+1\big)}r^n,\qquad S_{n-1}(r)=\frac{2\pi^{n/2}}{\Gamma(n/2)}r^{n-1}V[i][w] = max(V[i − 1][w], vᵢ + V[i − 1][w − wᵢ])
O(nW) dynamic programming.
Worked example: v = [60, 100, 120], w = [1, 2, 3], W = 5 → 220 ✓ checked
See also: longest common subsequence
V[i][w]=\max\big(V[i-1][w],\ v_i+V[i-1][w-w_i]\big)f = g + h
A consistent heuristic never needs to re-open a node.
Worked example: g = 7, h = 5 → 12 ✓ checked
See also: Dijkstra and Bellman–Ford relaxation
f(n)=g(n)+h(n),\quad h\ \text{admissible: }h(n)\le h^*(n)\ \Rightarrow\ \text{optimal}fundamental matrix N = (I − Q)⁻¹
where Q = transient-to-transient block, R = transient-to-absorbing
Worked example: Q = [[0, 0.5], [0.5, 0]] → [2, 2] ✓ checked
See also: stationary distribution of a Markov chain
N=(I-Q)^{-1},\quad \mathbb E[\text{steps to absorption}]=N\mathbf 1,\quad B=NRdoubling the capacity gives O(1) amortised appends
Worked example: n = 1000 → 1023 ✓ checked
See also: union–find complexity
\text{total copy cost for }n\text{ appends}\le2n,\qquad \text{amortised }O(1)≤ ⌈log₂(n + 1)⌉ comparisons
Worked example: n = 1e+06 → 20 ✓ checked
See also: comparison-sorting lower bound
\#\text{comparisons}\le\lceil\log_2(n+1)\rceilPascal's rule, row sum 2ⁿ, hockey stick, Vandermonde
Worked example: n = 10, k = 4 → [0, 0, 0] ✓ checked
See also: binomial theorem
\binom nk=\binom{n-1}{k-1}+\binom{n-1}k,\quad \sum_k\binom nk=2^n,\quad \sum_{i=k}^{n}\binom ik=\binom{n+1}{k+1},\quad \sum_k\binom mk\binom n{r-k}=\binom{m+n}rP = 1 − Π (365 − i)/365
23 people give about 50.7%.
Worked example: k = 23 → 0.507297 ✓ checked
See also: hash collisions (birthday bound)
P(\text{shared})=1-\prod_{i=0}^{k-1}\frac{365-i}{365}\approx1-e^{-k(k-1)/730}p ≈ (1 − e^(−kn/m))ᵏ; best k = (m/n) ln 2
where m bits, n items, k hashes
About 9.6 bits per item gives 1% false positives.
Worked example: m = 9.58506e+06, n = 1e+06, k = 7 → 0.0100392 ✓ checked
See also: hash collisions (birthday bound)
p\approx\Big(1-e^{-kn/m}\Big)^k,\qquad k^*=\frac mn\ln2,\qquad m=-\frac{n\ln p}{(\ln2)^2}Cₙ = C(2n, n)/(n + 1)
Counts balanced parentheses, binary trees with n nodes, and triangulations of an (n + 2)-gon.
Worked example: n = 5 → 42 ✓ checked
See also: binomial theorem
C_n=\frac1{n+1}\binom{2n}n,\qquad C_{n+1}=\sum_{i=0}^{n}C_iC_{n-i}unique solution mod N = Π nᵢ
Worked example: a = [2, 3, 2], n = [3, 5, 7] → 23 ✓ checked
See also: modular inverse and fast exponentiation
x\equiv a_i\pmod{n_i}\ (\gcd(n_i,n_j)=1)\ \Rightarrow\ x\equiv\sum_ia_iN_i\,(N_i^{-1}\bmod n_i)\pmod N,\ \ N_i=N/n_iany comparison sort needs ≥ log₂(n!) comparisons
Worked example: n = 1000 → [8529.4, 8523.08] ✓ checked
See also: Stirling's approximation
\#\text{comparisons}\ge\log_2(n!)\approx n\log_2n-1.44\,nhandshake lemma, complete graph, tree and Cayley's formula
Worked example: n = 6 → [15, 5, 1296] ✓ checked
See also: Euler's formula for planar graphs
\sum_v\deg v=2|E|,\quad |E_{K_n}|=\frac{n(n-1)}2,\quad \text{tree: }|E|=n-1,\quad \#\text{labelled trees}=n^{n-2}E[T] = n Hₙ
Collecting all 50 types takes about 225 draws.
Worked example: n = 50 → 224.96 ✓ checked
See also: harmonic numbers
\mathbb E[T]=nH_n\approx n\ln n+\gamma n,\qquad \operatorname{Var}(T)\le\frac{\pi^2}6n^2¬(A ∨ B) = ¬A ∧ ¬B; ¬(A ∧ B) = ¬A ∨ ¬B; |P(S)| = 2^|S|
Worked example: n = 5 → [32, 32] ✓ checked
See also: inclusion–exclusion
\overline{A\cup B}=\overline A\cap\overline B,\quad \overline{A\cap B}=\overline A\cup\overline B,\quad |\mathcal P(S)|=2^{|S|}Dₙ = n! Σ (−1)ᵏ/k! ≈ n!/e
The chance a random permutation has no fixed point tends to 1/e.
Worked example: n = 6 → 265 ✓ checked
See also: inclusion–exclusion
D_n=n!\sum_{k=0}^{n}\frac{(-1)^k}{k!}\approx\frac{n!}e,\qquad D_n=(n-1)(D_{n-1}+D_{n-2})relax each edge: d[v] = min(d[v], d[u] + w)
Dijkstra O((V + E) log V) for non-negative weights; Bellman–Ford O(VE) allows negative weights.
Worked example: W = [9 values] → [0, 3, 1] ✓ checked
See also: Floyd–Warshall
d[v]\leftarrow\min\big(d[v],\ d[u]+w(u,v)\big)number and sum of divisors from the prime factorisation
Worked example: n = 60 → [12, 168] ✓ checked
See also: Euler's totient function
\sigma_k(n)=\sum_{d\mid n}d^k,\quad \tau(n)=\prod(e_i+1)\ \text{for }n=\prod p_i^{e_i},\quad \sigma(n)=\prod\frac{p_i^{e_i+1}-1}{p_i-1}blocking probability of c servers with offered load a
Worked example: c = 3, a = 2 → 0.210526 ✓ checked
See also: M/M/1 queue
B(c,a)=\frac{a\,B(c-1,a)}{c+a\,B(c-1,a)},\quad B(0,a)=1gcd(a, b) = gcd(b, a mod b); lcm = ab/gcd
The extended algorithm finds x, y: the modular inverse when gcd = 1.
Worked example: a = 240, b = 46 → [2, 5520] ✓ checked
See also: modular inverse and fast exponentiation
\gcd(a,b)=\gcd(b,a\bmod b),\qquad ax+by=\gcd(a,b),\qquad \operatorname{lcm}(a,b)=\frac{ab}{\gcd(a,b)}V − E + F = 2; E ≤ 3V − 6
A cube has V = 8, E = 12, F = 6.
Worked example: V = 8, E = 12 → 6 ✓ checked
See also: counting in graphs
V-E+F=2,\qquad E\le3V-6\ \ (V\ge3)φ(n) = n Π (1 − 1/p) over primes p | n
Worked example: n = 36 → 12 ✓ checked
See also: Fermat's little theorem and Euler's theorem · RSA
\varphi(n)=n\prod_{p\mid n}\Big(1-\frac1p\Big),\qquad \varphi(pq)=(p-1)(q-1)a^(p−1) ≡ 1 (mod p); a^φ(n) ≡ 1 (mod n); Wilson: (p − 1)! ≡ −1
Worked example: a = 7, p = 13 → [1, 12] ✓ checked
See also: Euler's totient function · modular inverse and fast exponentiation
a^{p-1}\equiv1\pmod p,\qquad a^{\varphi(n)}\equiv1\pmod n\ \ (\gcd(a,n)=1),\qquad (p-1)!\equiv-1\pmod pFₙ = (φⁿ − ψⁿ)/√5
Worked example: n = 20 → 6765 ✓ checked
See also: linear recurrences
F_n=\frac{\varphi^n-\psi^n}{\sqrt5},\quad \varphi=\frac{1+\sqrt5}2,\ \psi=\frac{1-\sqrt5}2,\quad F_n=F_{n-1}+F_{n-2}all-pairs shortest paths in O(V³)
Worked example: W = [9 values] → 3 ✓ checked
See also: Dijkstra and Bellman–Ford relaxation
d_{ij}^{(k)}=\min\big(d_{ij}^{(k-1)},\ d_{ik}^{(k-1)}+d_{kj}^{(k-1)}\big)P(win) = (1 − (q/p)ⁱ)/(1 − (q/p)^N)
Worked example: p = 0.45, i = 10, N = 20 → 0.118501 ✓ checked
See also: random walk
P(\text{reach }N\text{ from }i)=\frac{1-(q/p)^i}{1-(q/p)^N}\ (p\ne q),\qquad \frac iN\ (p=q)expected collisions ≈ n²/(2m)
A 64-bit hash collides among about 5 billion items.
Worked example: n = 1000, m = 1e+06 → 0.4995 ✓ checked
See also: birthday problem
\mathbb E[\text{collisions}]=\binom n2\frac1m\approx\frac{n^2}{2m},\qquad n\approx\sqrt{2m\ln2}\ \text{for a 50\% chance}merge the two smallest weights until one remains
Worked example: p = [0.4, 0.3, 0.2, 0.1] → 1.9 ✓ checked
See also: source coding theorem · Kraft inequality
\text{repeatedly merge the two least probable symbols;}\quad \bar L=\sum_ip_i\ell_i,\ \ H\le\bar L<H+1HLL error 1.04/√m; count-min error e/w with probability 1 − e^(−d)
Worked example: m = 16384 → 0.008125 ✓ checked
See also: Bloom filter false-positive rate
\text{HLL standard error}\approx\frac{1.04}{\sqrt m},\qquad \text{CMS: }\varepsilon=\frac ew,\ \ \delta=e^{-d}|A ∪ B| = |A| + |B| − |A ∩ B|
Worked example: a = 30, b = 20, ab = 8 → 42 ✓ checked
See also: derangements
|A\cup B|=|A|+|B|-|A\cap B|,\qquad |A\cup B\cup C|=\sum|A_i|-\sum|A_i\cap A_j|+|A\cap B\cap C|f* = p − q/b
where p = win probability, q = 1 − p, b = net odds
Maximises the long-run growth rate of capital; half-Kelly is less volatile.
Worked example: p = 0.6, b = 1 → 0.2 ✓ checked
See also: gambler's ruin
f^*=p-\frac qb=\frac{bp-q}b,\qquad f^*=\frac{\mu-r}{\sigma^2}\ \text{(continuous)}solve the characteristic equation r² = c₁r + c₂
A repeated root r gives (A + Bn)rⁿ.
Worked example: c1 = 3, c2 = -2, n = 10 → 2047 ✓ checked
See also: Fibonacci numbers (Binet) · second-order linear ODE with constant coefficients
x_n=c_1x_{n-1}+c_2x_{n-2}\ \Rightarrow\ x_n=A r_1^n+Br_2^n,\quad r^2=c_1r+c_2dynamic programming over prefixes
ROUGE-L and diff tools use it.
Worked example: a = 'ABCBDAB', b = 'BDCABA' → 4 ✓ checked
See also: Levenshtein distance recurrence · 0/1 knapsack recurrence
L[i][j]=\begin{cases}L[i-1][j-1]+1&a_i=b_j\\ \max(L[i-1][j],L[i][j-1])&\text{otherwise}\end{cases}ρ = λ/μ; L = ρ/(1 − ρ); W = 1/(μ − λ) (Little's law L = λW)
where λ = arrival rate, μ = service rate
Waiting explodes as ρ → 1.
Worked example: lam = 8, mu = 10 → [0.8, 4, 0.5, 3.2] ✓ checked
See also: Erlang B
\rho=\frac\lambda\mu,\quad L=\frac\rho{1-\rho},\quad W=\frac1{\mu-\lambda},\quad L_q=\frac{\rho^2}{1-\rho},\quad L=\lambda Wcompare d with log_b a
Merge sort: a = 2, b = 2, d = 1 gives Θ(n log n).
Worked example: a = 2, b = 2, d = 1 → 0 ✓ checked
See also: FFT complexity
T(n)=aT(n/b)+\Theta(n^d)\ \Rightarrow\ T(n)=\begin{cases}\Theta(n^d)&d>\log_ba\\ \Theta(n^d\log n)&d=\log_ba\\ \Theta(n^{\log_ba})&d<\log_ba\end{cases}optimal parenthesisation by dynamic programming
(AB)C versus A(BC) can differ enormously in cost.
Worked example: p = [10, 30, 5, 60] → 4500 ✓ checked
See also: matrix-multiplication cost
m[i][j]=\min_{i\le k<j}\big(m[i][k]+m[k+1][j]+p_{i-1}p_kp_j\big)the value of a maximum flow equals the capacity of a minimum cut
Ford–Fulkerson / Edmonds–Karp find it by augmenting paths.
Worked example: caps = [3, 2, 4] → 3 ✓ checked
See also: Dijkstra and Bellman–Ford relaxation
\max\ \text{flow}=\min\ \text{cut capacity}a⁻¹ = a^(φ(n) − 1) mod n
Worked example: a = 3, n = 11 → [4, 1] ✓ checked
See also: Euclid's algorithm and Bézout · RSA
a^{-1}\equiv a^{\varphi(n)-1}\ (\bmod n),\qquad a^b\bmod n\ \text{by repeated squaring in }O(\log b)switching wins with probability 2/3
Bayes: P(car behind the other door | host opens a goat door) = 2/3.
Worked example: pa = 0.333333 → 0.666667 ✓ checked
See also: Bayes' theorem
P(\text{win by switching})=\frac23,\qquad P(\text{stay})=\frac13E[X₍ₖ₎] = k/(n + 1)
Worked example: n = 9, k = 3 → [0.3, 0.9, 0.0190909] ✓ checked
See also: continuous uniform distribution: mean
\mathbb E[X_{(k)}]=\frac k{n+1},\qquad \mathbb E[\max]=\frac n{n+1},\qquad \operatorname{Var}(X_{(k)})=\frac{k(n+1-k)}{(n+1)^2(n+2)}n!/(n₁! n₂! … n_k!)
MISSISSIPPI has 11!/(4!4!2!1!) = 34 650 arrangements.
Worked example: n = 11 → 34650 ✓ checked
See also: combinations and permutations
\frac{n!}{n_1!\,n_2!\cdots n_k!},\qquad n^k\ \text{(sequences of length }k\text{ from }n\text{ symbols)}some box holds ≥ ⌈n/m⌉ items
Worked example: n = 25, m = 12 → 3 ✓ checked
See also: birthday problem
n\text{ items in }m\text{ boxes}\ \Rightarrow\ \text{some box has at least }\Big\lceil\frac nm\Big\rceilπ(x) ≈ x/ln x
There are about 1/ln x primes near x.
Worked example: x = 100000 → 8685.89 ✓ checked
See also: Stirling's approximation
\pi(x)\sim\frac x{\ln x},\qquad p_n\sim n\ln nn = pq; e·d ≡ 1 (mod φ(n)); c = mᵉ mod n; m = cᵈ mod n
Security rests on the difficulty of factoring n.
Worked example: p = 61, q = 53, e = 17, m = 65 → [2753, 2790, 65] ✓ checked
See also: Euler's totient function · modular inverse and fast exponentiation
n=pq,\quad ed\equiv1\pmod{\varphi(n)},\quad c=m^e\bmod n,\quad m=c^d\bmod nreject the first n/e, then take the next record: success probability → 1/e
Worked example: n = 100 → [36.7879, 0.367879] ✓ checked
See also: birthday problem
\text{reject the first }n/e\text{ candidates, then take the next best;}\quad P(\text{best})\to\frac1e\approx0.368average marginal contribution over all orderings
The basis of SHAP feature attributions.
Worked example: v1 = 10, v2 = 20, v12 = 50 → [20, 30] ✓ checked
See also: mixed Nash equilibrium (2×2 zero-sum)
\varphi_i=\sum_{S\subseteq N\setminus\{i\}}\frac{|S|!\,(n-|S|-1)!}{n!}\big(v(S\cup\{i\})-v(S)\big)C(n + k − 1, k − 1)
Worked example: n = 10, k = 3 → [66, 36] ✓ checked
See also: combinations and permutations
\binom{n+k-1}{k-1}\ \text{(non-negative solutions of }x_1+\cdots+x_k=n),\qquad \binom{n-1}{k-1}\ \text{(positive)}S(n, k) = kS(n − 1, k) + S(n − 1, k − 1)
S(n, k) counts partitions of n items into k non-empty sets.
Worked example: n = 5 → [15, 52] ✓ checked
See also: Catalan numbers
S(n,k)=kS(n-1,k)+S(n-1,k-1),\qquad \text{Bell: }B_n=\sum_kS(n,k)geometric sum; compound interest (n times a year) and continuous compounding
Worked example: P = 1000, r = 0.05, t = 10, n = 12 → [1647.01, 1648.72] ✓ checked
See also: geometric series
S_n=a\frac{r^n-1}{r-1},\qquad A=P\Big(1+\frac rn\Big)^{nt},\qquad A=Pe^{rt}series multiplies, parallel multiplies the failure probabilities
Worked example: R = [0.9, 0.95], mtbf = 1000, mttr = 10 → [0.855, 0.995, 0.990099] ✓ checked
See also: hazard and survival functions
R_{\text{series}}=\prod_iR_i,\qquad R_{\text{parallel}}=1-\prod_i(1-R_i),\qquad A=\frac{\mathrm{MTBF}}{\mathrm{MTBF}+\mathrm{MTTR}}(n − 1)!/2 distinct tours
Worked example: n = 10 → 181440 ✓ checked
See also: combinations and permutations
\#\text{tours}=\frac{(n-1)!}2\ \ (\text{symmetric}),\qquad \text{Held–Karp: }O(n^22^n)nearly constant amortised time per operation
where α = inverse Ackermann
Worked example: n = 10 → 9 ✓ checked
See also: amortised cost of a doubling array
m\text{ operations on }n\text{ elements: }O\big(m\,\alpha(n)\big),\quad \alpha(n)\le4\ \text{for any practical }nY = a M^b; Kleiber's law B ∝ M^(3/4)
Linear in log–log coordinates: log Y = log a + b log M.
Worked example: M = 16 → 8 ✓ checked
See also: Pareto (type I) distribution: density
Y=aM^{b},\qquad \text{Kleiber: }B\propto M^{3/4}k = A e^(−Ea/RT)
Worked example: Ea = 50000, R = 8.314, T1 = 300, T2 = 310 → 1.90916 ✓ checked
See also: first-order kinetics and half-life
k=Ae^{-E_a/RT},\qquad \ln\frac{k_2}{k_1}=\frac{E_a}R\Big(\frac1{T_1}-\frac1{T_2}\Big)A = εcℓ = −log₁₀(I/I₀)
Worked example: eps = 1000, c = 0.001, l = 1 → [1, 0.1] ✓ checked
See also: pH and the Henderson–Hasselbalch equation
A=\varepsilon c\ell=-\log_{10}\frac I{I_0}p + ½ρv² + ρgh = const; A₁v₁ = A₂v₂; Re = ρvL/μ
Re ≲ 2000 laminar, ≳ 4000 turbulent in a pipe.
Worked example: rho = 1000, v = 2, L = 0.05, mu = 0.001 → 100000 ✓ checked
See also: Navier–Stokes (incompressible)
p+\tfrac12\rho v^2+\rho gh=\text{const},\qquad A_1v_1=A_2v_2,\qquad \mathrm{Re}=\frac{\rho vL}\mucall price and put–call parity
Worked example: S = 100, K = 100, r = 0.05, s = 0.2, T = 1 → 10.4506 ✓ checked
See also: normal cumulative distribution
C=S\,\Phi(d_1)-Ke^{-rT}\Phi(d_2),\quad d_{1,2}=\frac{\ln(S/K)+(r\pm\sigma^2/2)T}{\sigma\sqrt T},\quad C-P=S-Ke^{-rT}S = k ln W; pᵢ = e^(−Eᵢ/kT)/Z; F = −kT ln Z
The softmax is a Boltzmann distribution with temperature T; entropy S = −kΣ p ln p.
Worked example: E = [0, 1], kT = 1 → [0.731059, 0.268941] ✓ checked
See also: softmax · Shannon entropy · softmax with temperature
S=k_B\ln W,\qquad Z=\sum_ie^{-E_i/k_BT},\qquad p_i=\frac{e^{-E_i/k_BT}}Z,\qquad F=-k_BT\ln ZE[R] = r_f + β(E[R_m] − r_f)
Worked example: rf = 0.02, beta = 1.5, rm = 0.08 → 0.11 ✓ checked
See also: portfolio return, variance and Sharpe ratio
\mathbb E[R_i]=r_f+\beta_i\big(\mathbb E[R_m]-r_f\big),\qquad \mathrm{VaR}_\alpha=\mu+z_\alpha\sigmaF = kq₁q₂/r²; E = kq/r²; V = kq/r
Worked example: q1 = 1e-06, q2 = 1e-06, r = 0.1 → 0.898755 ✓ checked
See also: Newtonian gravitation
F=\frac{kq_1q_2}{r^2},\quad E=\frac{kq}{r^2},\quad V=\frac{kq}r,\quad k=\frac1{4\pi\varepsilon_0}\approx8.99\times10^9Shannon index H′ = −Σ pᵢ ln pᵢ; Simpson D = 1 − Σ pᵢ²
Shannon entropy by another name.
Worked example: p = [0.5, 0.3, 0.2] → [1.02965, 0.62] ✓ checked
See also: Shannon entropy · Gini impurity
H'=-\sum_ip_i\ln p_i,\qquad D=1-\sum_ip_i^2,\qquad \text{Hill number }{}^qD=\Big(\sum p_i^q\Big)^{\frac1{1-q}}t_d = ln 2/r ≈ 72/r%
Worked example: r = 0.07 → [9.9021, 10.2857] ✓ checked
See also: logistic growth
t_d=\frac{\ln2}r\approx\frac{72}{r\%},\qquad N=N_0e^{rt}ε = (dQ/dP)(P/Q); Y = A K^α L^(1 − α)
Worked example: A = 1, K = 8, L = 27, a = 0.333333 → 18 ✓ checked
See also: allometric scaling
\varepsilon=\frac{\partial\ln Q}{\partial\ln P}=\frac{dQ}{dP}\frac PQ,\qquad Y=AK^{\alpha}L^{1-\alpha}v = u + at; s = ut + ½at²; v² = u² + 2as
Worked example: u = 2, a = 3, t = 4 → [14, 32, 14] ✓ checked
See also: projectile motion
v=u+at,\qquad s=ut+\tfrac12at^2,\qquad v^2=u^2+2asU = W^(1−γ)/(1 − γ); certainty equivalent
Worked example: w = [50, 150], g = 0.5 → [93.3013, 6.69873] ✓ checked
See also: Jensen's inequality · Kelly criterion
U(W)=\frac{W^{1-\gamma}}{1-\gamma},\qquad \mathrm{CE}=U^{-1}\big(\mathbb E\,U(W)\big),\qquad \text{risk premium}=\mathbb E W-\mathrm{CE}occupation numbers of fermions and bosons
At E = μ the Fermi–Dirac occupation is ½.
Worked example: x = 1 → [0.268941, 0.581977] ✓ checked
See also: Boltzmann entropy and the partition function
f_{FD}(E)=\frac1{e^{(E-\mu)/k_BT}+1},\qquad f_{BE}(E)=\frac1{e^{(E-\mu)/k_BT}-1}ΔU = Q − W; Q = mcΔT; η = 1 − T_c/T_h
Worked example: Tc = 300, Th = 600, m = 2, c = 4186, dT = 10 → [0.5, 83720] ✓ checked
See also: Boltzmann entropy and the partition function
\Delta U=Q-W,\qquad Q=mc\Delta T,\qquad \eta_{\text{Carnot}}=1-\frac{T_c}{T_h}[A] = [A]₀ e^(−kt); t½ = ln 2/k
Also radioactive decay: N = N₀ e^(−λt).
Worked example: k = 0.1, t = 10 → [0.367879, 6.93147] ✓ checked
See also: Arrhenius equation
[A]=[A]_0e^{-kt},\qquad t_{1/2}=\frac{\ln2}kΔG = ΔH − TΔS; ΔG⁰ = −RT ln K
Worked example: dH = -50000, dS = -100, T = 300 → -20000 ✓ checked
See also: Nernst equation
\Delta G=\Delta H-T\Delta S,\qquad \Delta G^0=-RT\ln KG = ΣΣ|xᵢ − xⱼ|/(2n²x̄)
0 is perfect equality; 1 is maximal inequality. Also the Herfindahl index H = Σ sᵢ² for market concentration.
Worked example: x = [1, 2, 3, 4] → 0.25 ✓ checked
See also: Gini impurity
G=\frac{\sum_i\sum_j|x_i-x_j|}{2n^2\bar x}=1-2\int_0^1L(p)\,dpP = D₁/(r − g)
Requires r > g.
Worked example: D1 = 2, r = 0.08, g = 0.03 → 40 ✓ checked
See also: net present value and annuities
P=\frac{D_1}{r-g}genotype frequencies p², 2pq, q²
Worked example: q = 0.3 → [0.49, 0.42, 0.09] ✓ checked
See also: binomial distribution
p^2+2pq+q^2=1,\qquad p+q=1heat u_t = αu_xx; wave u_tt = c²u_xx; Laplace ∇²u = 0
Separation of variables turns each into ordinary differential equations. The heat kernel is a Gaussian with variance 2αt.
Worked example: a = 1, t = 0.5, x = 1 → 6.45248e-09 ✓ checked
See also: Navier–Stokes (incompressible)
u_t=\alpha\,u_{xx},\qquad u_{tt}=c^2u_{xx},\qquad \nabla^2u=0F = −kx; ω = √(k/m); T = 2π√(m/k)
Worked example: k = 100, m = 1 → [10, 0.628319] ✓ checked
See also: simple pendulum
F=-kx,\quad U=\tfrac12kx^2,\quad \omega=\sqrt{k/m},\quad T=2\pi\sqrt{m/k},\quad x(t)=A\cos(\omega t+\phi)PV = nRT; v_rms = √(3kT/m)
where R = 8.314 J/(mol K)
Worked example: n = 1, T = 273.15, P = 101325 → 0.022414 ✓ checked
See also: Maxwell–Boltzmann speeds
PV=nRT=Nk_BT,\qquad v_{\text{rms}}=\sqrt{\frac{3k_BT}m},\qquad \bar K=\tfrac32k_BTτ dV/dt = −(V − V_rest) + RI
The basic unit of spiking neural networks.
Worked example: tau = 10, R = 10, I = 2, t = 10 → 12.6424 ✓ checked
See also: RC and RL circuits
\tau_m\frac{dV}{dt}=-(V-V_{\text{rest}})+RI,\qquad \text{spike when }V\ge V_{\text{th}},\ \text{then reset}dN/dt = rN(1 − N/K); N(t) = K/(1 + ((K − N₀)/N₀) e^(−rt))
Exponential growth N₀e^(rt) while N ≪ K; the sigmoid shape of adoption curves.
Worked example: K = 1000, N0 = 10, r = 0.5, t = 10 → 599.86 ✓ checked
See also: sigmoid (logistic function) · doubling time and the rule of 72
\frac{dN}{dt}=rN\Big(1-\frac NK\Big),\qquad N(t)=\frac K{1+\frac{K-N_0}{N_0}e^{-rt}}F = q(E + v × B); gyroradius r = mv/(qB)
Worked example: q = 1.6e-19, v = [1e+06, 0, 0], B = [0, 0, 1] → [0, -1.6e-13, 0] ✓ checked
See also: cross product and triple product
\mathbf F=q\big(\mathbf E+\mathbf v\times\mathbf B\big),\qquad r=\frac{mv}{qB},\qquad \omega_c=\frac{qB}mprey x, predators y; equilibrium (γ/δ, α/β)
Worked example: a = 1, b = 0.1, d = 0.02, g = 0.4 → [20, 10] ✓ checked
See also: logistic growth
\dot x=\alpha x-\beta xy,\quad \dot y=\delta xy-\gamma y,\qquad (x^*,y^*)=\Big(\frac\gamma\delta,\frac\alpha\beta\Big)Gauss, Gauss for magnetism, Faraday, Ampère–Maxwell
The speed of light is c = 1/√(μ₀ε₀).
Worked example: mu0 = 1.25664e-06, e0 = 8.85419e-12 → 2.99792e+08 ✓ checked
See also: gradient, divergence and curl
\nabla\cdot\mathbf E=\frac\rho{\varepsilon_0},\quad \nabla\cdot\mathbf B=0,\quad \nabla\times\mathbf E=-\frac{\partial\mathbf B}{\partial t},\quad \nabla\times\mathbf B=\mu_0\mathbf J+\mu_0\varepsilon_0\frac{\partial\mathbf E}{\partial t}most probable, mean and rms speed
Worked example: kT = 4.14e-21, m = 4.65e-26 → [421.977, 476.15, 516.814] ✓ checked
See also: Maxwell–Boltzmann distribution: mean · ideal gas law and kinetic theory
v_p=\sqrt{\frac{2k_BT}m},\quad \bar v=\sqrt{\frac{8k_BT}{\pi m}},\quad v_{\text{rms}}=\sqrt{\frac{3k_BT}m}v = Vmax[S]/(Km + [S]); Hill: θ = [L]ⁿ/(K_dⁿ + [L]ⁿ)
Half the maximum rate at [S] = Km.
Worked example: Vmax = 10, Km = 2, S = 2 → 5 ✓ checked
See also: sigmoid (logistic function)
v=\frac{V_{\max}[S]}{K_m+[S]},\qquad \theta=\frac{[L]^n}{K_d^n+[L]^n}momentum balance plus incompressibility
Also the setting of physics-informed neural networks (PINNs).
Worked example: d = 1 → 1 ✓ checked
See also: Bernoulli's equation and continuity · heat, wave and Laplace equations
\rho\Big(\frac{\partial\mathbf u}{\partial t}+(\mathbf u\cdot\nabla)\mathbf u\Big)=-\nabla p+\mu\nabla^2\mathbf u+\mathbf f,\qquad \nabla\cdot\mathbf u=0E = E⁰ − (RT/nF) ln Q
In neuroscience it gives the equilibrium potential of an ion.
Worked example: R = 8.314, T = 310, z = 1, F = 96485, out = 5, in_ = 140 → -0.089011 ✓ checked
See also: Gibbs free energy and equilibrium constant
E=E^0-\frac{RT}{nF}\ln Q,\qquad E_{\text{ion}}=\frac{RT}{zF}\ln\frac{[\mathrm{ion}]_{out}}{[\mathrm{ion}]_{in}}NPV = Σ Cₜ/(1 + r)ᵗ; annuity PV = C(1 − (1 + r)⁻ⁿ)/r
Worked example: C = [-100, 60, 60], r = 0.1 → 4.13223 ✓ checked
See also: sum of a geometric sequence and compound growth
\mathrm{NPV}=\sum_{t=0}^{T}\frac{C_t}{(1+r)^t},\qquad PV_{\text{annuity}}=C\,\frac{1-(1+r)^{-n}}r,\qquad \text{perpetuity }\frac CrF = ma = dp/dt; impulse = Δp
Worked example: m = 2, a = 3, dt = 5 → [6, 30] ✓ checked
See also: work, kinetic and potential energy
\mathbf F=m\mathbf a=\frac{d\mathbf p}{dt},\qquad \mathbf p=m\mathbf v,\qquad \mathbf J=\int\mathbf F\,dt=\Delta\mathbf pF = Gm₁m₂/r²; v_orb = √(GM/r); v_esc = √(2GM/r)
where G = 6.674×10⁻¹¹ N m²/kg²
Worked example: G = 6.674e-11, M = 5.972e+24, r = 6.371e+06 → [7909.5, 11185.7] ✓ checked
See also: work, kinetic and potential energy
F=\frac{Gm_1m_2}{r^2},\quad v_{\text{orb}}=\sqrt{\frac{GM}r},\quad v_{\text{esc}}=\sqrt{\frac{2GM}r},\quad T^2=\frac{4\pi^2a^3}{GM}V = IR; P = IV; series add, parallel add reciprocals
Worked example: R = [100, 200, 300], V = 12 → [600, 54.5455, 0.02] ✓ checked
See also: RC and RL circuits
V=IR,\quad P=IV=I^2R,\quad R_{\text{series}}=\sum R_i,\quad \frac1{R_{\text{par}}}=\sum\frac1{R_i}Eₙ = n²h²/(8mL²); hydrogen Eₙ = −13.6 eV/n²
Worked example: n = 2 → [-3.4, 10.2] ✓ checked
See also: photon energy and de Broglie wavelength
E_n=\frac{n^2h^2}{8mL^2},\qquad E_n^{\mathrm H}=-\frac{13.6\ \text{eV}}{n^2}pH = −log₁₀[H⁺]; pH = pKa + log₁₀([A⁻]/[HA])
Worked example: h = 0.001, pka = 4.76, ratio = 10 → [3, 5.76] ✓ checked
See also: Arrhenius equation
\mathrm{pH}=-\log_{10}[\mathrm H^+],\qquad \mathrm{pH}=\mathrm{p}K_a+\log_{10}\frac{[\mathrm A^-]}{[\mathrm{HA}]}E = hf = hc/λ; λ = h/p; ΔxΔp ≥ ħ/2
where h = 6.626×10⁻³⁴ J s
Worked example: h = 6.62607e-34, c = 2.99792e+08, lam = 5e-07 → 3.97289e-19 ✓ checked
See also: time–frequency uncertainty principle
E=hf=\frac{hc}\lambda,\qquad \lambda=\frac hp,\qquad \Delta x\,\Delta p\ge\frac\hbar2blackbody spectrum; P = σAT⁴; λ_max = b/T
where σ = 5.670×10⁻⁸, b = 2.898×10⁻³ m K
Worked example: T = 5772 → [5.02039e-07, 6.29386e+07] ✓ checked
See also: photon energy and de Broglie wavelength
B_\lambda=\frac{2hc^2}{\lambda^5}\frac1{e^{hc/\lambda k_BT}-1},\qquad P=\sigma AT^4,\qquad \lambda_{\max}=\frac bTμ_p = wᵀμ; σ_p² = wᵀΣw; Sharpe = (μ_p − r_f)/σ_p
Diversification: the covariance terms, not the variances, set the portfolio risk.
Worked example: w = [0.5, 0.5], mu = [0.08, 0.12], S = [[0.04, 0.01], [0.01, 0.09]], rf = 0.02 → [0.1, 0.0375, 0.413118] ✓ checked
See also: variance and standard deviation · covariance
\mu_p=\mathbf w^{\top}\boldsymbol\mu,\qquad \sigma_p^2=\mathbf w^{\top}\Sigma\mathbf w,\qquad \mathrm{Sharpe}=\frac{\mu_p-r_f}{\sigma_p}range v²sin 2θ/g; height v²sin²θ/(2g); time 2v sin θ/g
Maximum range at 45° (no drag).
Worked example: v = 20, th = 0.785398, g = 9.80665 → [40.7886, 10.1972, 2.88419] ✓ checked
See also: equations of constant acceleration
R=\frac{v_0^2\sin2\theta}g,\qquad H=\frac{v_0^2\sin^2\theta}{2g},\qquad T=\frac{2v_0\sin\theta}gV(t) = V₀(1 − e^(−t/RC)); τ = RC
63% charged after one time constant.
Worked example: V0 = 5, R = 1000, C = 1e-06, t = 0.001 → 3.1606 ✓ checked
See also: Ohm's law and resistor networks · first-order low-pass filter
V_C(t)=V_0\big(1-e^{-t/RC}\big),\quad \tau=RC,\quad E_C=\tfrac12CV^2,\quad E_L=\tfrac12LI^2τ = Iα; L = Iω; K = ½Iω²
Worked example: m = 2, r = 0.5, w = 10 → [0.25, 12.5, 2.5] ✓ checked
See also: work, kinetic and potential energy
\tau=I\alpha,\quad L=I\omega,\quad K_{\text{rot}}=\tfrac12I\omega^2,\quad I_{\text{disk}}=\tfrac12mr^2,\quad I_{\text{sphere}}=\tfrac25mr^2iħ ∂ψ/∂t = Ĥψ; Ĥψ = Eψ
|ψ|² is the probability density.
Worked example: n = 1, L = 1 → 1 ✓ checked
See also: particle in a box and hydrogen levels
i\hbar\frac{\partial\psi}{\partial t}=\hat H\psi,\qquad \hat H\psi=E\psi,\quad \hat H=-\frac{\hbar^2}{2m}\nabla^2+VT = 2π√(L/g)
Worked example: L = 1, g = 9.80665 → 2.00641 ✓ checked
See also: Hooke's law and the spring-mass oscillator
T=2\pi\sqrt{\frac Lg}\quad(\text{small angles})R₀ = β/γ; herd-immunity threshold 1 − 1/R₀
An outbreak grows iff R₀ S/N > 1.
Worked example: beta = 0.3, gamma = 0.1 → [3, 0.666667] ✓ checked
See also: logistic growth
\dot S=-\beta SI,\quad \dot I=\beta SI-\gamma I,\quad \dot R=\gamma I,\qquad R_0=\frac\beta\gamma,\quad \text{herd immunity }1-\frac1{R_0}E = mc²; γ = 1/√(1 − v²/c²); time dilation, length contraction
Worked example: b = 0.6 → [1.25, 12.5, 8] ✓ checked
See also: Lorentz force
E=mc^2,\quad \gamma=\frac1{\sqrt{1-v^2/c^2}},\quad \Delta t=\gamma\Delta t_0,\quad L=\frac{L_0}\gamma,\quad E^2=(pc)^2+(mc^2)^2a = v²/r = ω²r
Worked example: v = 10, r = 5, m = 2 → [20, 40] ✓ checked
See also: Newton's second law, momentum and impulse
a_c=\frac{v^2}r=\omega^2r,\qquad v=\omega r,\qquad F_c=\frac{mv^2}rv = fλ; Doppler; Snell; thin lens
Worked example: f = 440, v = 343, vs = 30 → [0.779545, 482.173] ✓ checked
See also: Maxwell's equations
v=f\lambda,\quad f'=f\frac{v\pm v_o}{v\mp v_s},\quad n_1\sin\theta_1=n_2\sin\theta_2,\quad \frac1f=\frac1{d_o}+\frac1{d_i}W = Fd cos θ; K = ½mv²; U = mgh
Work–energy theorem: W = ΔK.
Worked example: m = 2, v = 3, h = 10 → [9, 196.133] ✓ checked
See also: Newton's second law, momentum and impulse
W=Fd\cos\theta,\quad K=\tfrac12mv^2,\quad U=mgh,\quad P=\frac{dW}{dt}=\mathbf F\cdot\mathbf vgain in decibels = 20 log₁₀|G|
Worked example: g = 0.1 → -20 ✓ checked
See also: signal-to-noise ratio in decibels · Butterworth magnitude response
|G(j\omega)|_{\text{dB}}=20\log_{10}|G(j\omega)|,\qquad \text{GM}=\frac1{|G(j\omega_{180})|}full-rank controllability / observability matrix
Worked example: A = [[0, 1], [0, 0]], B = [[0], [1]] → 2 ✓ checked
See also: linear-quadratic regulator
\operatorname{rank}\big[B,\ AB,\ \dots,\ A^{n-1}B\big]=n,\qquad \operatorname{rank}\big[C;\ CA;\ \dots;\ CA^{n-1}\big]=nfirst order 1 − e^(−t/τ); second order: overshoot exp(−πζ/√(1 − ζ²))
Worked example: z = 0.5, wn = 2 → [0.163034, 4] ✓ checked
See also: RC and RL circuits · second-order linear ODE with constant coefficients
y(t)=1-e^{-t/\tau},\qquad \text{overshoot}=\exp\!\Big(-\frac{\pi\zeta}{\sqrt{1-\zeta^2}}\Big),\quad t_s\approx\frac4{\zeta\omega_n}optimal feedback gain from the algebraic Riccati equation
Scalar case: P = (a + √(a² + b²q/r))·r/b².
Worked example: a = 1, b = 1, q = 1, r = 1 → [2.41421, 2.41421, 0] ✓ checked
See also: Kalman filter
\min\int(\mathbf x^{\top}Q\mathbf x+\mathbf u^{\top}R\mathbf u)dt\ \Rightarrow\ \mathbf u=-K\mathbf x,\ K=R^{-1}B^{\top}P,\ \ A^{\top}P+PA-PBR^{-1}B^{\top}P+Q=0stable iff every eigenvalue of A has negative real part
Worked example: A = [[-1, 0], [0, -2]] → 1 ✓ checked
See also: eigenvalue equation
\dot{\mathbf x}=A\mathbf x\ \text{stable}\iff\exists P\succ0:\ A^{\top}P+PA=-Q\ (Q\succ0)u = Kp·e + Ki·∫e + Kd·de/dt
Worked example: Kp = 2, Ki = 0.5, Kd = 0.1, e = 1, ie = 4, de = -2 → 3.8 ✓ checked
See also: first- and second-order step responses
u(t)=K_pe(t)+K_i\int_0^te\,d\tau+K_d\frac{de}{dt}forward Euler is stable for |1 + hλ| < 1
For λ = −100 the step must be below 0.02. Stiff problems need implicit methods.
Worked example: lam = -100, h = 0.019 → 0.9 ✓ checked
See also: Euler's method · gradient descent step
y'=\lambda y:\ \ y_{n+1}=(1+h\lambda)y_n,\quad \text{stable iff }|1+h\lambda|<1f(xₖ) − f* ≤ 2L‖x₀ − x*‖²/(k + 1)²
Optimal for first-order methods on smooth convex functions; plain gradient descent only gets O(1/k).
Worked example: L = 2, R2 = 1, k = 9 → 0.04 ✓ checked
See also: Nesterov accelerated gradient
f(x_k)-f^*\le\frac{2L\|x_0-x^*\|^2}{(k+1)^2}Δθ = −√(E[Δθ²] + ε)/√(E[g²] + ε) · g
No learning rate needed.
Worked example: g = 2, rho = 0.9, eps = 1e-06 → -0.00316227 ✓ checked
E[g^2]\leftarrow\rho E[g^2]+(1-\rho)g^2,\quad \Delta\theta=-\frac{\sqrt{E[\Delta\theta^2]+\epsilon}}{\sqrt{E[g^2]+\epsilon}}\,g,\quad E[\Delta\theta^2]\leftarrow\rho E[\Delta\theta^2]+(1-\rho)\Delta\theta^2G ← G + g²; θ ← θ − η g / (√G + ε)
Per-parameter step shrinks as gradients accumulate; good for sparse features.
Worked example: G0 = 0, g = 2, theta = 1, lr = 0.1, eps = 1e-12 → 0.9 ✓ checked
G\leftarrow G+g^2,\quad \theta\leftarrow\theta-\frac{\eta\,g}{\sqrt G+\epsilon}m ← β₁m + (1−β₁)g v ← β₂v + (1−β₂)g² m̂ = m/(1−β₁ᵗ) v̂ = v/(1−β₂ᵗ) θ ← θ − η m̂ / (√v̂ + ε)
where defaults β₁ = 0.9, β₂ = 0.999, ε = 1e-8
The bias correction matters in the first steps.
Worked example: g = 2, t = 1, theta = 1, lr = 0.001, b1 = 0.9, b2 = 0.999, eps = 1e-08 → 0.999 ✓ checked
See also: AdamW (decoupled weight decay) · momentum (heavy ball) · RMSProp
m\leftarrow\beta_1m+(1-\beta_1)g,\ v\leftarrow\beta_2v+(1-\beta_2)g^2,\ \hat m=\tfrac{m}{1-\beta_1^t},\ \hat v=\tfrac{v}{1-\beta_2^t},\ \theta\leftarrow\theta-\eta\frac{\hat m}{\sqrt{\hat v}+\epsilon}θ ← θ − η ( m̂/(√v̂+ε) + λθ )
where λ = weight decay
Decay is applied to the weights, not folded into the gradient.
Worked example: g = 2, t = 1, theta = 1, lr = 0.001, b1 = 0.9, b2 = 0.999, eps = 1e-08, wd = 0.01 → 0.99899 ✓ checked
See also: Adam · L2 penalty (weight decay)
\theta\leftarrow\theta-\eta\left(\frac{\hat m}{\sqrt{\hat v}+\epsilon}+\lambda\theta\right)x-update (ridge solve), z-update (soft-threshold), u-update (dual)
Splits a hard problem into easy ones; good for distributed and constrained problems.
Worked example: A = [[1, 0], [0, 1]], b = [3, 0.2], lam = 0.5, rho = 1 → [1, 0, 0.5, 0.1] ✓ checked
See also: ISTA (proximal gradient for the lasso) · proximal operator of the L1 norm
x\leftarrow(A^{\top}A+\rho I)^{-1}\big(A^{\top}b+\rho(z-u)\big),\quad z\leftarrow S_{\lambda/\rho}(x+u),\quad u\leftarrow u+x-zv̂ₜ = max(v̂ₜ₋₁, vₜ)
Keeps the second-moment estimate non-decreasing, fixing a convergence gap in Adam.
Worked example: vprev = 4, v = 1 → 4 ✓ checked
See also: Adam
\hat v_t=\max(\hat v_{t-1},v_t),\quad \theta\leftarrow\theta-\eta\frac{m_t}{\sqrt{\hat v_t}+\epsilon}f(x + tp) ≤ f(x) + c₁ t ∇fᵀp
where c₁ ≈ 10⁻⁴
Backtracking: shrink t by a factor until it holds.
Worked example: x = 1, p = -2, t = 0.5, c = 0.0001 → 1 ✓ checked
See also: Wolfe curvature condition
f(x+tp)\le f(x)+c_1\,t\,\nabla f(x)^{\top}pH₊ = (I − ρsyᵀ) H (I − ρysᵀ) + ρssᵀ
where s = x₊ − x, y = ∇f₊ − ∇f
Satisfies the secant equation H₊y = s.
Worked example: H = [[1, 0], [0, 1]], s = [1, 0.5], y = [2, 1] → [1, 0.5] ✓ checked
See also: Newton's method (multivariate)
H_{k+1}=\Big(I-\rho s y^{\top}\Big)H_k\Big(I-\rho y s^{\top}\Big)+\rho ss^{\top},\quad \rho=\frac1{y^{\top}s}convex: f(θx + (1−θ)y) ≤ θf(x) + (1−θ)f(y); μ-strongly convex: f(y) ≥ f(x) + ∇f·(y−x) + (μ/2)‖y − x‖²
Worked example: x = 1, y = 3, t = 0.25 → 0.75 ✓ checked
See also: descent lemma (L-smoothness)
f(\theta x+(1-\theta)y)\le\theta f(x)+(1-\theta)f(y);\quad f(y)\ge f(x)+\nabla f(x)^{\top}(y-x)+\tfrac\mu2\|y-x\|^2η_t = η_min + ½(η_max − η_min)(1 + cos(πT_cur/T_i))
where T_cur = steps since the last restart
Worked example: hi = 0.1, lo = 0, tcur = 25, ti = 100 → 0.0853553 ✓ checked
See also: cosine learning-rate schedule
\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\Big(1+\cos\frac{\pi T_{\text{cur}}}{T_i}\Big)η_t = η_min + ½(η_max − η_min)(1 + cos(πt/T))
where T = total steps
Worked example: hi = 0.1, lo = 0, t = 50, T = 100 → 0.05 ✓ checked
See also: linear learning-rate warm-up · exponential learning-rate decay
\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\left(1+\cos\frac{\pi t}{T}\right)f(y) ≤ f(x) + ∇f(x)·(y−x) + (L/2)‖y−x‖²
where L = Lipschitz constant of ∇f
Gives the 1/L step size for gradient descent.
Worked example: fx = 0, gr = 0, L = 2, d = 1 → 1 ✓ checked
See also: gradient descent rate (strongly convex) · gradient descent step
f(y)\le f(x)+\nabla f(x)^{\top}(y-x)+\frac L2\|y-x\|^2xᵢ ← xᵢ e^(−ηgᵢ) / Σⱼ xⱼ e^(−ηgⱼ)
Mirror descent with the KL divergence; keeps x on the simplex. The multiplicative-weights update.
Worked example: x = [0.5, 0.5], g = [1, 0], eta = 1 → [0.268941, 0.731059] ✓ checked
See also: projected gradient descent · regret of online gradient descent
x_i\leftarrow\frac{x_i\,e^{-\eta g_i}}{\sum_jx_je^{-\eta g_j}}sort v descending as u; ρ as above; subtract τ and clip
Used to keep probabilities valid in constrained optimisation.
Worked example: v = [0.5, 1.2, -0.3] → [0.15, 0.85, 0] ✓ checked
See also: projected gradient descent
\Pi(v)=\max(v-\tau,0),\quad \tau=\frac{\sum_{i\le\rho}u_i-1}{\rho},\ \ \rho=\max\Big\{j:u_j-\frac{\sum_{i\le j}u_i-1}j>0\Big\}η_t = η₀ γᵗ
where 0 < γ < 1
Worked example: lr0 = 0.1, gamma = 0.9, t = 2 → 0.081 ✓ checked
See also: cosine learning-rate schedule
\eta_t=\eta_0\,\gamma^{t}θ̄ ← β θ̄ + (1−β) θ
where β ≈ 0.99–0.9999
Used for weights (EMA models) and running statistics.
Worked example: avg = 1, theta = 2, beta = 0.99 → 1.01 ✓ checked
See also: momentum (heavy ball)
\bar\theta\leftarrow\beta\bar\theta+(1-\beta)\thetatₖ₊₁ = (1 + √(1 + 4tₖ²))/2; yₖ₊₁ = xₖ + ((tₖ − 1)/tₖ₊₁)(xₖ − xₖ₋₁)
Worked example: t = 1 → 1.61803 ✓ checked
See also: ISTA (proximal gradient for the lasso) · accelerated gradient rate
t_{k+1}=\frac{1+\sqrt{1+4t_k^2}}2,\qquad y_{k+1}=x_k+\frac{t_k-1}{t_{k+1}}(x_k-x_{k-1})s = argmin ⟨∇f, s⟩; x ← (1 − γ)x + γs
Projection-free: only a linear minimisation over the constraint set. On the simplex s is a vertex.
Worked example: g = [3, 1, 2], x = [0.333333, 0.333333, 0.333333], k = 0 → [0, 1, 0] ✓ checked
See also: projected gradient descent
s_k=\arg\min_{s\in\mathcal C}\langle\nabla f(x_k),s\rangle,\qquad x_{k+1}=(1-\gamma_k)x_k+\gamma_ks_k,\ \gamma_k=\frac2{k+2}Δ = −(JᵀJ)⁻¹Jᵀr; LM adds λI
where r = residuals, J = their Jacobian
Exact in one step for linear least squares; LM interpolates toward gradient descent.
Worked example: J = [[1, 0], [1, 1], [1, 2]], y = [1, 3, 5], x = [0, 0] → [1, 2] ✓ checked
See also: ordinary least squares (normal equations) · Newton's method (multivariate)
\Delta=-(J^{\top}J)^{-1}J^{\top}r,\qquad \Delta_{LM}=-(J^{\top}J+\lambda I)^{-1}J^{\top}raverage K micro-batch gradients before stepping
Gives a large effective batch on small memory; scale each loss by 1/K.
Worked example: K = 8, B = 32 → 256 ✓ checked
See also: learning-rate scaling with batch size
g=\frac1K\sum_{k=1}^{K}\nabla L_{B_k},\qquad B_{\text{effective}}=K\cdot Brelative error ‖g_a − g_n‖/max(‖g_a‖, ‖g_n‖) ≲ 1e-6
Compare backprop with the central difference.
Worked example: ga = [1, 2], gn = [1, 2] → 6.32455e-08 ✓ checked
See also: finite differences
\frac{\|g_{\text{analytic}}-g_{\text{numeric}}\|}{\max(\|g_{\text{analytic}}\|,\|g_{\text{numeric}}\|)}\ \lesssim 10^{-6}‖x_k − x*‖² ≤ (1 − μ/L)ᵏ ‖x₀ − x*‖²
where μ-strongly convex, L-smooth, step η = 1/L; μ/L = 1/κ
Linear convergence; rate is governed by the condition number.
Worked example: mu = 1, L = 4, k = 2, d0 = 1 → 0.5625 ✓ checked
See also: condition number · descent lemma (L-smoothness)
\|x_k-x^*\|^2\le\left(1-\frac{\mu}{L}\right)^{k}\|x_0-x^*\|^2θ ← θ − η ∇L(θ)
where η = learning rate
Converges for convex L-smooth objectives when η ≤ 1/L.
Worked example: theta = [1, 2], g = [0.5, -0.5], lr = 0.1 → [0.95, 2.05] ✓ checked
See also: momentum (heavy ball) · Adam · gradient descent rate (strongly convex)
\theta_{t+1}=\theta_t-\eta\,\nabla L(\theta_t)B_noise = tr(Σ)/‖G‖²
where Σ = per-example gradient covariance, G = true gradient
The batch size beyond which larger batches stop helping (critical batch size).
Worked example: trS = 100, G2 = 0.25 → 400 ✓ checked
See also: learning-rate scaling with batch size
\mathcal B_{\text{noise}}=\frac{\operatorname{tr}(\Sigma)}{\|G\|^2}g ← g · min(1, c / ‖g‖₂)
where c = clip threshold
Rescales the whole gradient, keeping its direction.
Worked example: g = [3, 4], c = 1 → [0.6, 0.8] ✓ checked
See also: gradient descent step
g\leftarrow g\cdot\min\!\left(1,\frac{c}{\|g\|_2}\right)Hv ≈ (∇f(x + εv) − ∇f(x))/ε
Needs only gradients; reverse-mode autodiff computes it exactly in about twice the cost of a gradient.
Worked example: A = [[2, 1], [1, 3]], x = [1, 1], v = [1, 0], e = 1e-06 → [2, 1] ✓ checked
See also: Hessian matrix and second-derivative test · conjugate gradient method
H\mathbf v\approx\frac{\nabla f(x+\varepsilon\mathbf v)-\nabla f(x)}{\varepsilon}=\nabla_x\big(\nabla f(x)^{\top}\mathbf v\big)x ← S_{ηλ}(x − ηAᵀ(Ax − b))FISTA adds Nesterov momentum for an O(1/k²) rate.
Worked example: A = [[1, 0], [0, 1]], b = [3, 0.2], x = [0, 0], lam = 0.5, eta = 1 → [2.5, 0] ✓ checked
See also: FISTA momentum · proximal operator of the L1 norm
x\leftarrow\operatorname{prox}_{\eta\lambda\|\cdot\|_1}\big(x-\eta A^{\top}(Ax-b)\big)∇f + Σ μᵢ∇gᵢ = 0; μᵢ ≥ 0; gᵢ ≤ 0; μᵢgᵢ = 0
where gᵢ(x) ≤ 0 constraints, μᵢ multipliers
Complementary slackness: a constraint is active or its multiplier is zero.
Worked example: mu = 0.5, g = 0 → 0 ✓ checked
See also: Lagrange multipliers · support vector machine (soft margin)
\nabla f(x)+\sum_i\mu_i\nabla g_i(x)=0,\quad \mu_i\ge0,\quad g_i(x)\le0,\quad \mu_i g_i(x)=0θ ← (1 − ηλ)θ − η∇L
Identical for plain SGD; not for Adam, which is why AdamW decouples it.
Worked example: th = 1, g = 0.5, lr = 0.1, lam = 0.01 → 0.949 ✓ checked
See also: AdamW (decoupled weight decay) · L2 penalty (weight decay)
\theta\leftarrow\theta-\eta\big(\nabla L+\lambda\theta\big)=(1-\eta\lambda)\theta-\eta\nabla LL(x,λ) = f(x) + λ g(x); ∇ₓL = 0, g(x) = 0
where equality constraint g(x) = 0
Example: min x²+y² subject to x+y=1 gives x=y=½, λ=−1.
Worked example: x = 0.5, lam = -1 → 0 ✓ checked
See also: KKT stationarity (inequality constraints)
\mathcal L(x,\lambda)=f(x)+\lambda\,g(x),\quad \nabla_x\mathcal L=0,\ g(x)=0g(λ) = inf_x L(x, λ); d* ≤ p* (weak duality)
Strong duality (d* = p*) holds for convex problems under Slater's condition. Example: min x² s.t. x ≥ 1 has g(λ) = λ − λ²/4, maximised at λ = 2.
Worked example: lam = 2 → 1 ✓ checked
See also: KKT stationarity (inequality constraints) · SVM dual problem
g(\lambda)=\inf_x\ \mathcal L(x,\lambda),\quad d^*=\max_{\lambda\ge0}g(\lambda)\le p^*λ_ℓ = η‖θ_ℓ‖/(‖g_ℓ‖ + β‖θ_ℓ‖)
Scales each layer's step to its weight norm; enables very large batches.
Worked example: w = 10, g = 2, b = 0, lr = 0.1 → 0.5 ✓ checked
See also: AdamW (decoupled weight decay) · learning-rate scaling with batch size
\lambda_\ell=\eta\,\frac{\|\theta_\ell\|}{\|g_\ell\|+\beta\|\theta_\ell\|},\qquad \theta_\ell\leftarrow\theta_\ell-\lambda_\ell\big(g_\ell+\beta\theta_\ell\big)η′ = η·B′/B
Valid up to a critical batch size; combine with warm-up.
Worked example: lr = 0.1, B = 256, B2 = 1024 → [0.4, 0.2] ✓ checked
See also: gradient noise scale
\eta'=\eta\,\frac{B'}{B}\ \ (\text{SGD}),\qquad \eta'=\eta\sqrt{\frac{B'}{B}}\ \ (\text{Adam, common heuristic})η_t = η_max · t / T_w
where T_w = warm-up steps
Stabilises early training of large models.
Worked example: lr = 0.001, t = 100, tw = 1000 → 0.0001 ✓ checked
See also: cosine learning-rate schedule
\eta_t=\eta_{\max}\,\frac{t}{T_w}\quad(t\le T_w)u = sign(β₁m + (1−β₁)g); θ ← θ − η(u + λθ)
Sign updates make every coordinate move by the same magnitude; found by program search.
Worked example: m = 0.5, g = -2, b1 = 0.9, th = 1, lr = 0.01, wd = 0.1 → 0.989 ✓ checked
See also: AdamW (decoupled weight decay)
\mathbf u=\operatorname{sign}\big(\beta_1m+(1-\beta_1)g\big),\quad \theta\leftarrow\theta-\eta(\mathbf u+\lambda\theta),\quad m\leftarrow\beta_2m+(1-\beta_2)gslow ← slow + α(fast − slow) every k steps
Worked example: slow = 1, fast = 2, a = 0.5 → 1.5 ✓ checked
See also: exponential moving average
\phi\leftarrow\phi+\alpha(\theta_k-\phi)v ← μv + g; θ ← θ − ηv
where μ ≈ 0.9 = momentum coefficient
Accumulates a velocity that smooths noisy gradients.
Worked example: v = 0.1, g = 0.5, mu = 0.9, theta = 1, lr = 0.1 → 0.941 ✓ checked
See also: gradient descent step · Adam
v\leftarrow\mu v+g,\quad \theta\leftarrow\theta-\eta vv ← μv − η∇f(θ + μv); θ ← θ + v
Evaluates the gradient at the look-ahead point; O(1/k²) for smooth convex f.
Worked example: th = 1, v = 0.1, mu = 0.9, lr = 0.1 → [0.981, -0.019] ✓ checked
See also: momentum (heavy ball) · accelerated gradient rate
\mathbf v\leftarrow\mu\mathbf v-\eta\nabla f(\theta+\mu\mathbf v),\quad \theta\leftarrow\theta+\mathbf v½‖∇f‖² ≥ μ(f − f*) ⇒ linear rate (1 − μ/L)ᵏ
Linear convergence without convexity; holds for many over-parameterised networks.
Worked example: mu = 1, L = 4, k = 3, gap0 = 8 → 3.375 ✓ checked
See also: gradient descent rate (strongly convex)
\tfrac12\|\nabla f(x)\|^2\ge\mu\big(f(x)-f^*\big)\ \Rightarrow\ f(x_k)-f^*\le\Big(1-\frac\mu L\Big)^k\big(f(x_0)-f^*\big)η_t = (η₀ − η_end)(1 − t/T)ᵖ + η_end
p = 1 is linear decay.
Worked example: lr0 = 0.1, end = 0, t = 50, T = 100, p = 2 → 0.025 ✓ checked
See also: cosine learning-rate schedule
\eta_t=(\eta_0-\eta_{\text{end}})\Big(1-\frac tT\Big)^p+\eta_{\text{end}}x ← Π_C(x − η∇f(x))
where Π_C = Euclidean projection onto C
For a box [l, u] the projection is a clip.
Worked example: x = [0.9, 0.1], g = [-5, 2], lr = 0.1 → [1, 0] ✓ checked
See also: Frank–Wolfe (conditional gradient) · Euclidean projection onto the simplex
x\leftarrow\Pi_{\mathcal C}\big(x-\eta\nabla f(x)\big)prox(v) = sign(v)(|v| − λ)₊ (soft-thresholding)
Defined in general as argmin f(x) + (1/2λ)‖x − v‖².
Worked example: v = [3, -0.5, 1.2], lam = 1 → [2, -0, 0.2] ✓ checked
See also: lasso solution for an orthonormal design · ISTA (proximal gradient for the lasso)
\operatorname{prox}_{\lambda\|\cdot\|_1}(v)=\operatorname{sign}(v)\,(|v|-\lambda)_+Regret_T ≤ (3/2) R G √T
Hedge / exponentiated gradient: regret ≤ √(T ln N / 2) for N experts.
Worked example: T = 10000, N = 10 → [107.298] ✓ checked
See also: entropic mirror descent (exponentiated gradient) · UCB1
\operatorname{Regret}_T=\sum_{t=1}^{T}f_t(x_t)-\min_x\sum_tf_t(x)\le\frac{3RG}2\sqrt Ts ← βs + (1−β)g²; θ ← θ − η g / (√s + ε)
where β ≈ 0.9
Worked example: s0 = 0, g = 2, beta = 0.9, theta = 1, lr = 0.1, eps = 1e-12 → 0.683772 ✓ checked
s\leftarrow\beta s+(1-\beta)g^2,\quad\theta\leftarrow\theta-\frac{\eta\,g}{\sqrt s+\epsilon}E f(x̄_T) − f* ≤ R G/√T
where R = ‖x₀ − x*‖, G = bound on gradient norm
Worked example: R = 2, G = 3, T = 10000 → 0.06 ✓ checked
See also: subgradient method bound
\mathbb E\big[f(\bar x_T)\big]-f^*\le\frac{R\,G}{\sqrt T}\quad\big(\eta=\tfrac{R}{G\sqrt T}\big)ε = ρ∇L/‖∇L‖; w ← w − η∇L(w + ε)
where ρ = neighbourhood radius
Seeks flat minima by taking the gradient at the worst nearby point.
Worked example: w = 1, rho = 0.1, lr = 0.1 → 0.89 ✓ checked
See also: gradient descent step
\varepsilon=\rho\frac{\nabla L(w)}{\|\nabla L(w)\|},\qquad w\leftarrow w-\eta\,\nabla L(w+\varepsilon)η_t = η₀ γ^⌊t/s⌋
Worked example: lr0 = 0.1, gamma = 0.1, s = 30, t = 65 → 0.001 ✓ checked
See also: exponential learning-rate decay
\eta_t=\eta_0\,\gamma^{\lfloor t/s\rfloor}min f(xᵢ) − f* ≤ R²/(2αk) + αG²/2
where constant step size α
Worked example: R = 2, G = 1, al = 0.1, k = 100 → 0.25 ✓ checked
See also: SGD rate (convex, bounded gradients)
\min_{i\le k}f(x_i)-f^*\le\frac{R^2}{2\alpha k}+\frac{\alpha G^2}{2}max Σαᵢ − ½ ΣΣ αᵢαⱼyᵢyⱼ k(xᵢ, xⱼ)
Only support vectors have αᵢ > 0. Replace the dot product by a kernel.
Worked example: al = 0.5 → 0.5 ✓ checked
See also: support vector machine (soft margin) · RBF (Gaussian) kernel
\max_\alpha\ \sum_i\alpha_i-\tfrac12\sum_{i,j}\alpha_i\alpha_jy_iy_j\,k(\mathbf x_i,\mathbf x_j)\quad\text{s.t. }0\le\alpha_i\le C,\ \sum_i\alpha_iy_i=0η_t = d^(−½) min(t^(−½), t·t_w^(−3/2))
Linear warm-up for t_w steps, then decay as 1/√t ("Attention Is All You Need").
Worked example: d = 512, tw = 4000, t = 1000 → 0.000174693 ✓ checked
See also: linear learning-rate warm-up
\eta_t=d_{\text{model}}^{-1/2}\min\big(t^{-1/2},\ t\,t_w^{-3/2}\big)ρ = actual reduction / predicted reduction
where m = local quadratic model
ρ near 1: model is good, enlarge the region; ρ small: shrink it.
Worked example: act = 0.8, pred = 1 → 0.8 ✓ checked
See also: Newton's method (multivariate)
\rho=\frac{f(x)-f(x+p)}{m(0)-m(p)}Var(ĝ_B) = Σ/B
Halving the noise needs four times the batch.
Worked example: s = 4, B = 64 → 0.0625 ✓ checked
See also: gradient noise scale
\operatorname{Var}\big(\hat g_B\big)=\frac{\Sigma}{B}\quad\text{(sampling with replacement)}∇f(x + tp)ᵀp ≥ c₂ ∇f(x)ᵀp
Together with Armijo it rules out too-short steps; required for BFGS.
Worked example: x = 1, p = -2, t = 0.1, c2 = 0.9 → 1 ✓ checked
See also: Armijo (sufficient decrease) condition
\nabla f(x+tp)^{\top}p\ge c_2\,\nabla f(x)^{\top}p,\quad 0<c_1<c_2<1ℓ = 1[y ≠ ŷ]; risk = error rate
Non-differentiable, so surrogates (log, hinge, exponential) replace it in training.
Worked example: y = [0, 1, 1, 0], yh = [0, 1, 0, 0] → 0.25 ✓ checked
See also: Bayes error and optimal classifier
\ell(y,\hat y)=\mathbf 1[y\ne\hat y],\qquad R(h)=\mathbb E\,\ell(y,h(x))R²_adj = 1 − (1−R²)(n−1)/(n−p−1)
where n samples, p predictors
Penalises adding useless predictors.
Worked example: r2 = 0.97, n = 20, p = 3 → 0.964375 ✓ checked
See also: coefficient of determination R²
R^2_{\mathrm{adj}}=1-(1-R^2)\frac{n-1}{n-p-1}AUC = P(score⁺ > score⁻) + ½ P(tie)
where s⁺, s⁻ = scores of a random positive and negative
Threshold-free ranking quality; 0.5 is chance.
Worked example: pos = [0.8, 0.6], neg = [0.5, 0.6] → 0.875 ✓ checked
See also: recall
\mathrm{AUC}=P\big(s^{+}>s^{-}\big)+\tfrac12P\big(s^{+}=s^{-}\big)AP = Σ (Rₙ − Rₙ₋₁) Pₙ
where precision and recall at each threshold
Preferred to ROC-AUC for heavy class imbalance.
Worked example: labels = [1, 0, 1, 0, 1] → 0.755556 ✓ checked
See also: area under the ROC curve (AUC)
\mathrm{AP}=\sum_n(R_n-R_{n-1})\,P_nBA = (TPR + TNR)/2
Accuracy that does not reward predicting the majority class.
Worked example: tp = 8, fn = 2, tn = 70, fp = 20 → 0.788889 ✓ checked
See also: recall · specificity, NPV, FPR and FNR
\mathrm{BA}=\tfrac12\Big(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\Big)L = −[ y log p + (1−y) log(1−p) ]
where y ∈ {0,1}, p = predicted probability of class 1
Worked example: y = 1, p = 0.8 → 0.223144 ✓ checked
See also: cross-entropy · gradient of logistic loss
L=-\big[y\log p+(1-y)\log(1-p)\big]L = max(z, 0) − zy + ln(1 + e^(−|z|))
where z = logit
Algebraically equal to −y ln σ(z) − (1 − y) ln(1 − σ(z)) without overflow.
Worked example: z = 2, y = 1 → 0.126928 ✓ checked
See also: binary cross-entropy (log loss) · stable softmax and log-sum-exp
L=\max(z,0)-zy+\ln\big(1+e^{-|z|}\big)BP = 1 if c > r, else e^(1 − r/c)
where c = candidate length, r = reference length
Worked example: c = 8, r = 10 → 0.778801 ✓ checked
See also: BLEU score
\mathrm{BP}=\begin{cases}1&c>r\\ e^{1-r/c}&c\le r\end{cases}BLEU = BP · exp( Σ wₙ log pₙ )
where pₙ = n-gram precision, r/c = reference/candidate length
Worked example: p1 = 0.8, p2 = 0.6 → 0.69282 ✓ checked
See also: perplexity
\mathrm{BLEU}=\mathrm{BP}\cdot\exp\!\Big(\sum_{n=1}^{N}w_n\log p_n\Big),\quad \mathrm{BP}=\min\big(1,e^{1-r/c}\big)BS = (1/n) Σᵢ Σₖ (pᵢₖ − 1[k = yᵢ])²
A proper scoring rule: lowest in expectation only when the probabilities are honest.
Worked example: p = [[0.8, 0.2], [0.3, 0.7]], y = [0, 1] → 0.13 ✓ checked
See also: expected calibration error (ECE) · binary cross-entropy (log loss)
\mathrm{BS}=\frac1n\sum_i\sum_k\big(p_{ik}-\mathbf{1}[k=y_i]\big)^2BS = REL − RES + UNC; UNC = ō(1 − ō)
where ō = base rate
REL is the calibration error part: lower is better.
Worked example: o = 0.3 → 0.21 ✓ checked
See also: Brier score · expected calibration error (ECE)
\mathrm{BS}=\underbrace{\mathrm{REL}}_{\text{reliability}}-\underbrace{\mathrm{RES}}_{\text{resolution}}+\underbrace{\mathrm{UNC}}_{\text{uncertainty}},\quad \mathrm{UNC}=\bar o(1-\bar o)CH = (SS_B/(k − 1)) / (SS_W/(n − k))
The ANOVA F ratio applied to clusters; higher is better.
Worked example: ssb = 40, ssw = 10, k = 3, n = 33 → 60 ✓ checked
See also: one-way ANOVA F statistic
\mathrm{CH}=\frac{\mathrm{SS}_B/(k-1)}{\mathrm{SS}_W/(n-k)}accuracy = (TP + TN)/N; prevalence = (TP + FN)/N
Misleading when classes are imbalanced: always predicting the majority class can score highly.
Worked example: tp = 40, tn = 45, fp = 5, fn = 10 → [0.85, 0.5, 0.15] ✓ checked
See also: balanced accuracy · specificity, NPV, FPR and FNR · precision · recall
\mathrm{Acc}=\frac{TP+TN}{TP+TN+FP+FN},\qquad \text{prevalence}=\frac{TP+FN}Npurity = (1/n) Σ_k max_j |c_k ∩ t_j|
Biased toward many small clusters.
Worked example: m = [8, 7], n = 20 → 0.75 ✓ checked
See also: normalised mutual information
\mathrm{purity}=\frac1n\sum_k\max_j|c_k\cap t_j|R² = 1 − SS_res / SS_tot
1 is perfect; can be negative for a bad model.
Worked example: y = [1, 2, 3], yh = [1.1, 1.9, 3.2] → 0.97 ✓ checked
See also: adjusted R² · mean squared error
R^2=1-\frac{\sum(y_i-\hat y_i)^2}{\sum(y_i-\bar y)^2}L = y d² + (1 − y) max(0, m − d)²
where y = 1 for similar pairs
Worked example: d = 0.4, y = 0, m = 1 → 0.36 ✓ checked
See also: triplet loss
L=y\,d^2+(1-y)\max(0,\,m-d)^2DB = (1/k) Σᵢ maxⱼ≠ᵢ (sᵢ + sⱼ)/dᵢⱼ
where s = within-cluster scatter, d = centroid distance
Lower is better.
Worked example: s = [1, 1], d = 4 → 0.5 ✓ checked
See also: silhouette coefficient
\mathrm{DB}=\frac1k\sum_i\max_{j\ne i}\frac{s_i+s_j}{d_{ij}}DCG = Σ (2^relᵢ − 1)/log₂(i + 1); NDCG = DCG/IDCG
Rewards relevant items near the top.
Worked example: rel = [3, 2, 3, 0] → 0.959454 ✓ checked
See also: mean reciprocal rank
\mathrm{DCG}@k=\sum_{i=1}^{k}\frac{2^{rel_i}-1}{\log_2(i+1)},\qquad \mathrm{NDCG}@k=\frac{\mathrm{DCG}@k}{\mathrm{IDCG}@k}Dice = 2|A ∩ B| / (|A| + |B|)
Worked example: inter = 3, a = 5, b = 5 → 0.6 ✓ checked
See also: intersection over union (Jaccard)
\mathrm{Dice}=\frac{2|A\cap B|}{|A|+|B|}L = 1 − (2Σpᵢgᵢ + ε)/(Σpᵢ + Σgᵢ + ε)
Directly optimises overlap; standard for segmentation with small foregrounds.
Worked example: p = [0.9, 0.1, 0.8], g = [1, 0, 1] → 0.105263 ✓ checked
See also: Dice coefficient · Tversky index
L=1-\frac{2\sum p_ig_i+\epsilon}{\sum p_i+\sum g_i+\epsilon}H(mean p) = mean H(pₘ) + mutual information
Epistemic uncertainty is the disagreement between ensemble members.
Worked example: P = [[0.9, 0.1], [0.1, 0.9]] → [0.693147, 0.325083, 0.368064] ✓ checked
See also: Shannon entropy · mutual information
\underbrace{H\big(\bar p\big)}_{\text{total}}=\underbrace{\mathbb E_m H(p_m)}_{\text{aleatoric}}+\underbrace{I(y;m)}_{\text{epistemic}},\quad \bar p=\frac1M\sum_mp_mECE = Σ_b (n_b/n) · |acc(b) − conf(b)|
where B confidence bins, n_b examples in bin b, acc(b) = fraction correct, conf(b) = mean confidence
Measures whether stated confidence matches how often the model is right. Depends on the binning; 15 bins is common.
Worked example: conf = [0.9, 0.9, 0.6, 0.6], ok = [1, 0, 1, 1], B = 10 → 0.4 ✓ checked
See also: Brier score · softmax with temperature · maximum calibration error (MCE)
\mathrm{ECE}=\sum_{b=1}^{B}\frac{n_b}{n}\,\big|\operatorname{acc}(b)-\operatorname{conf}(b)\big|EV = 1 − Var(y − ŷ)/Var(y)
Like R² but ignores a constant bias in the residuals.
Worked example: y = [1, 2, 3], yh = [1.5, 2.5, 3.5] → 1 ✓ checked
See also: coefficient of determination R²
\mathrm{EV}=1-\frac{\operatorname{Var}(y-\hat y)}{\operatorname{Var}(y)}L = exp(−y f(x))
where y ∈ {−1, +1}
The loss AdaBoost minimises.
Worked example: y = 1, f = 0.5 → 0.606531 ✓ checked
See also: AdaBoost
L=e^{-yf(x)}F₁ = 2PR / (P + R)
where P = precision, R = recall
Harmonic mean of precision and recall.
Worked example: p = 0.8, r = 0.666667 → 0.727273 ✓ checked
See also: precision · recall · Fβ score
F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN}FL = −α (1 − pₜ)^γ ln pₜ
where pₜ = probability of the true class, γ ≈ 2
Down-weights easy examples so rare hard ones dominate; used for dense detection.
Worked example: p = 0.9, g = 2, al = 0.25 → 0.000263401 ✓ checked
See also: binary cross-entropy (log loss)
\mathrm{FL}(p_t)=-\alpha_t(1-p_t)^{\gamma}\ln p_tFID = ‖μ₁ − μ₂‖² + tr(Σ₁ + Σ₂ − 2(Σ₁Σ₂)^½)
where Gaussian fits to Inception features of real and generated images
Lower is better. It is the squared Wasserstein-2 distance between the two Gaussians.
Worked example: m1 = [0, 0], m2 = [1, 0], S1 = [[1, 0], [0, 1]], S2 = [[4, 0], [0, 1]] → 2 ✓ checked
See also: Wasserstein-2 distance between Gaussians (1-D)
\mathrm{FID}=\|\mu_1-\mu_2\|^2+\operatorname{tr}\big(\Sigma_1+\Sigma_2-2(\Sigma_1\Sigma_2)^{1/2}\big)F_β = (1+β²) P R / (β² P + R)
where β > 1 favours recall, β < 1 favours precision
Worked example: p = 0.8, r = 0.666667, b = 2 → 0.689655 ✓ checked
See also: F1 score
F_\beta=\frac{(1+\beta^2)\,PR}{\beta^2P+R}L = ½ ln(2πσ²) + (y − μ)²/(2σ²)
where the network predicts both μ and σ
Learns its own uncertainty; MSE is the case of fixed σ.
Worked example: y = 1, mu = 0, s = 2 → 1.73709 ✓ checked
See also: mean squared error · normal (Gaussian) density
L=\tfrac12\ln(2\pi\sigma^2)+\frac{(y-\mu)^2}{2\sigma^2}GELU(x) = x Φ(x)
where Φ = standard normal cdf
The activation in BERT and GPT-style transformers.
Worked example: x = 1 → 0.841345 ✓ checked
See also: ReLU and leaky ReLU · normal cumulative distribution · Swish / SiLU
\operatorname{GELU}(x)=x\,\Phi(x)=\tfrac{x}{2}\left[1+\operatorname{erf}\!\left(\tfrac{x}{\sqrt2}\right)\right]L = max(0, 1 − y f(x))
where y ∈ {−1,+1}
The SVM loss.
Worked example: y = 1, f = 0.3 → 0.7 ✓ checked
See also: support vector machine (soft margin)
L=\max\big(0,\,1-y\,f(x)\big)L(a) = ½a² if |a| ≤ δ, else δ(|a| − ½δ)
where a = residual y − ŷ
Quadratic near zero, linear in the tails.
Worked example: a = 3, d = 1 → 2.5 ✓ checked
See also: mean squared error · mean absolute error
L_\delta(a)=\begin{cases}\tfrac12a^2&|a|\le\delta\\ \delta\big(|a|-\tfrac12\delta\big)&|a|>\delta\end{cases}tanh x = (eˣ − e⁻ˣ)/(eˣ + e⁻ˣ)
Zero-centred, range (−1, 1).
Worked example: x = 1 → 0.761594 ✓ checked
See also: sigmoid (logistic function)
\tanh x=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}=2\sigma(2x)-1IS = exp(E KL(p(y|x) ‖ p(y)))
High when each image is classified confidently and the classes are diverse.
Worked example: P = [[1, 0], [0, 1]] → 2 ✓ checked
See also: Fréchet inception distance
\mathrm{IS}=\exp\Big(\mathbb E_x\,D_{\mathrm{KL}}\big(p(y\mid x)\,\|\,p(y)\big)\Big)IoU = |A ∩ B| / |A ∪ B|
Standard for detection and segmentation.
Worked example: inter = 3, a = 5, b = 5 → 0.428571 ✓ checked
See also: Dice coefficient
\mathrm{IoU}=\frac{|A\cap B|}{|A\cup B|}average the error over k held-out folds
k = 5 or 10 is typical; leave-one-out is k = n.
Worked example: e = [0.1, 0.12, 0.08, 0.11, 0.09] → [0.1, 0.00707107] ✓ checked
See also: leave-one-out residual shortcut · bootstrap standard error
\widehat{\mathrm{Err}}=\frac1k\sum_{i=1}^{k}\mathrm{Err}_i,\qquad \mathrm{SE}\approx\frac{s}{\sqrt k}L = τ² KL(softmax(z_t/τ) ‖ softmax(z_s/τ))
where t = teacher, s = student, τ = temperature
The τ² factor keeps gradient magnitudes comparable as τ changes.
Worked example: zt = [2, 0], zs = [1, 0], tau = 2 → 0.105378 ✓ checked
See also: Kullback–Leibler divergence · softmax with temperature
L=\tau^2\,D_{\mathrm{KL}}\big(\operatorname{softmax}(z_t/\tau)\,\|\,\operatorname{softmax}(z_s/\tau)\big)y′ₖ = (1−ε)·1[k=y] + ε/K
where K classes, ε ≈ 0.1
Softens one-hot targets to discourage overconfidence.
Worked example: eps = 0.1, K = 4, y = 0 → [0.925, 0.025, 0.025, 0.025] ✓ checked
See also: softmax cross-entropy (from logits)
y'_k=(1-\varepsilon)\,\mathbf{1}[k=y]+\frac{\varepsilon}{K}dᵢⱼ = min(dᵢ₋₁,ⱼ + 1, dᵢ,ⱼ₋₁ + 1, dᵢ₋₁,ⱼ₋₁ + 1[aᵢ ≠ bⱼ])
Dynamic programming, O(mn).
Worked example: a = 'kitten', b = 'sitting' → 3 ✓ checked
See also: word error rate
d_{i,j}=\min\big(d_{i-1,j}+1,\ d_{i,j-1}+1,\ d_{i-1,j-1}+\mathbf 1[a_i\ne b_j]\big)LR⁺ = TPR/FPR; LR⁻ = FNR/TNR
Multiply the prior odds by LR to get the posterior odds.
Worked example: tpr = 0.8, fpr = 0.1 → [8, 0.222222] ✓ checked
See also: Bayes factor and posterior odds · specificity, NPV, FPR and FNR
\mathrm{LR}^+=\frac{\mathrm{TPR}}{\mathrm{FPR}},\quad \mathrm{LR}^-=\frac{\mathrm{FNR}}{\mathrm{TNR}},\quad \mathrm{DOR}=\frac{\mathrm{LR}^+}{\mathrm{LR}^-}L = ln cosh(ŷ − y)
Behaves like ½a² near 0 and |a| − ln 2 in the tails; smooth everywhere.
Worked example: a = 2 → 1.325 ✓ checked
See also: Huber loss
L=\ln\cosh(\hat y-y)LSE(z) = m + log Σ exp(zⱼ − m)
A smooth maximum: max(z) ≤ LSE(z) ≤ max(z) + log n.
Worked example: z = [0, 0] → 0.693147 ✓ checked
See also: softmax
\operatorname{LSE}(z)=\log\sum_j e^{z_j}=m+\log\sum_j e^{z_j-m},\ m=\max_j z_jMAPE = (100/n) Σ |(y − ŷ)/y|
MAPE explodes when y ≈ 0 and penalises over-forecasts less than under-forecasts.
Worked example: y = [100, 200], yh = [110, 180] → [10, 10.0251] ✓ checked
See also: mean absolute error
\mathrm{MAPE}=\frac{100}n\sum\Big|\frac{y-\hat y}y\Big|,\qquad \mathrm{sMAPE}=\frac{100}n\sum\frac{2|y-\hat y|}{|y|+|\hat y|}L = max(0, −y(x₁ − x₂) + m)
where y = +1 if x₁ should rank higher
Worked example: x1 = 0.3, x2 = 0.5, y = 1, m = 0.2 → 0.4 ✓ checked
See also: hinge loss
L=\max\big(0,\,-y(x_1-x_2)+m\big)MCC = (TP·TN − FP·FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN))
Balanced even for imbalanced classes; ranges −1 to 1.
Worked example: tp = 8, fp = 2, fn = 4, tn = 6 → 0.408248 ✓ checked
See also: F1 score
\mathrm{MCC}=\frac{TP\cdot TN-FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}MCE = max_b |acc(b) − conf(b)|
where same bins as ECE
The worst bin, however few examples it holds.
Worked example: gaps = [0.4, 0.4, 0.05] → 0.4 ✓ checked
See also: expected calibration error (ECE)
\mathrm{MCE}=\max_{b}\big|\operatorname{acc}(b)-\operatorname{conf}(b)\big|MAE = (1/n) Σ |yᵢ − ŷᵢ|
More robust to outliers than MSE.
Worked example: y = [1, 2, 3], yh = [1, 2, 5] → 0.666667 ✓ checked
See also: mean squared error · Huber loss
\mathrm{MAE}=\frac1n\sum_i|y_i-\hat y_i|MASE = MAE / (MAE of the naive one-step forecast)
Below 1 beats the naive forecast; scale-free.
Worked example: y = [10, 12, 11, 15], yh = [11, 11, 12, 14] → 0.428571 ✓ checked
See also: MAPE and sMAPE
\mathrm{MASE}=\frac{\frac1n\sum|y_t-\hat y_t|}{\frac1{n-1}\sum|y_t-y_{t-1}|}MRR = mean(1/rank of the first relevant result)
Worked example: ranks = [1, 3, 2] → 0.611111 ✓ checked
See also: DCG and NDCG
\mathrm{MRR}=\frac1{|Q|}\sum_{q}\frac1{\mathrm{rank}_q}MSE = (1/n) Σ (yᵢ − ŷᵢ)²
Sensitive to outliers; RMSE = √MSE is in the units of y.
Worked example: y = [1, 2, 3], yh = [1, 2, 5] → 1.33333 ✓ checked
See also: mean absolute error · root mean squared error · bias–variance decomposition
\mathrm{MSE}=\frac1n\sum_i(y_i-\hat y_i)^2L = Σ_{j≠y} max(0, sⱼ − s_y + Δ)where s = class scores
Worked example: s = [3.2, 5.1, -1.7], y = 0, d = 1 → 2.9 ✓ checked
See also: hinge loss
L=\sum_{j\ne y}\max\big(0,\ s_j-s_y+\Delta\big)NMI = 2I(U;V)/(H(U) + H(V))
Agreement between two clusterings or a clustering and labels, in [0, 1].
Worked example: I = 0.5, hu = 0.69, hv = 0.69 → 0.724638 ✓ checked
See also: mutual information · Rand index and adjusted Rand index
\mathrm{NMI}=\frac{2\,I(U;V)}{H(U)+H(V)}L = μ − y ln μ + ln y!
where μ = predicted rate, y = observed count
The loss for count data; μ = exp(model output).
Worked example: mu = 3, y = 2 → 1.49592 ✓ checked
See also: Poisson regression log-likelihood
L=\mu-y\ln\mu+\ln y!Precision = TP / (TP + FP)
where TP/FP = true/false positives
Of the items flagged, how many were right.
Worked example: tp = 8, fp = 2 → 0.8 ✓ checked
\mathrm{Precision}=\frac{TP}{TP+FP}P@k, R@k
Worked example: rel = [1, 0, 1, 1, 0], k = 3, total = 4 → [0.666667, 0.5] ✓ checked
See also: average precision (PR-AUC)
P@k=\frac{|\text{relevant in top }k|}k,\qquad R@k=\frac{|\text{relevant in top }k|}{|\text{relevant}|}RI = (agreeing pairs)/C(n, 2)
where a = pairs together in both, b = apart in both
ARI is 0 for random labelings and 1 for identical ones.
Worked example: a = 4, b = 8, n = 6 → 0.8 ✓ checked
See also: normalised mutual information
\mathrm{RI}=\frac{a+b}{\binom n2},\qquad \mathrm{ARI}=\frac{\mathrm{RI}-\mathbb E[\mathrm{RI}]}{\max\mathrm{RI}-\mathbb E[\mathrm{RI}]}Recall = TP / (TP + FN)
where FN = false negatives
Of the real positives, how many were found. Also called sensitivity or TPR.
Worked example: tp = 8, fn = 4 → 0.666667 ✓ checked
See also: precision · F1 score
\mathrm{Recall}=\frac{TP}{TP+FN}ReLU(x) = max(0, x)
where α ≈ 0.01
Gradient is 1 for x > 0, 0 for x < 0.
Worked example: x = -2, a = 0.01 → -0.02 ✓ checked
\operatorname{ReLU}(x)=\max(0,x),\qquad \operatorname{LeakyReLU}(x)=\max(\alpha x,x)RMSE = √( (1/n) Σ (yᵢ − ŷᵢ)² )
Worked example: y = [1, 2, 3], yh = [1, 2, 5] → 1.1547 ✓ checked
See also: mean squared error
\mathrm{RMSE}=\sqrt{\frac1n\sum_i(y_i-\hat y_i)^2}RMSLE = √(mean((ln(1 + ŷ) − ln(1 + y))²))
Cares about relative, not absolute, error.
Worked example: y = [3, 5], yh = [2, 6] → 0.230785 ✓ checked
See also: root mean squared error
\mathrm{RMSLE}=\sqrt{\frac1n\sum\big(\ln(1+\hat y)-\ln(1+y)\big)^2}ROUGE-N = overlapping n-grams / n-grams in the reference (recall)
ROUGE-L uses the longest common subsequence.
Worked example: overlap = 4, ref = 6 → 0.666667 ✓ checked
See also: BLEU score · F1 score
\mathrm{ROUGE\text{-}N}=\frac{\sum_{g\in\text{ref}}\min\big(\mathrm{count}_{\text{cand}}(g),\mathrm{count}_{\text{ref}}(g)\big)}{\sum_{g\in\text{ref}}\mathrm{count}_{\text{ref}}(g)}σ(x) = 1 / (1 + e⁻ˣ); σ′ = σ(1 − σ)
Saturates for large |x|, so gradients vanish.
Worked example: x = 0 → 0.5 ✓ checked
See also: hyperbolic tangent · softplus · logistic regression model
\sigma(x)=\frac{1}{1+e^{-x}},\qquad \sigma'(x)=\sigma(x)\big(1-\sigma(x)\big)s = (b − a)/max(a, b)
where a = mean distance to own cluster, b = mean distance to the nearest other cluster
Between −1 and 1; higher is better.
Worked example: a = 1, b = 3 → 0.666667 ✓ checked
See also: k-means objective
s=\frac{b-a}{\max(a,b)}L = ½a²/β if |a| < β, else |a| − ½β
Detection models (Faster R-CNN) use it for box regression.
Worked example: a = 0.5, b = 1 → 0.125 ✓ checked
See also: Huber loss
L=\begin{cases}\tfrac12a^2/\beta&|a|<\beta\\ |a|-\tfrac12\beta&\text{otherwise}\end{cases}softmax(z)ₖ = exp(zₖ) / Σⱼ exp(zⱼ)
Subtract max(z) before exponentiating for stability.
Worked example: z = [0, 0] → [0.5, 0.5] ✓ checked
See also: softmax cross-entropy (from logits) · softmax with temperature · log-sum-exp
\operatorname{softmax}(z)_k=\frac{e^{z_k}}{\sum_j e^{z_j}}L = −z_y + log Σⱼ exp(zⱼ)
where z = logits, y = true class
The numerically stable form of −log softmax(z)_y.
Worked example: z = [2, 0], y = 0 → 0.126928 ✓ checked
See also: softmax · log-sum-exp · gradient of softmax cross-entropy
L=-z_y+\log\sum_j e^{z_j}softmax(z/T)ₖ
where T > 1 flattens, T < 1 sharpens
Used for calibration and sampling.
Worked example: z = [1, 0], T = 2 → 0.622459 ✓ checked
See also: softmax
\operatorname{softmax}(z/T)_k=\frac{e^{z_k/T}}{\sum_j e^{z_j/T}}softplus(x) = ln(1 + eˣ)
A smooth approximation of ReLU; its derivative is the sigmoid.
Worked example: x = 0 → 0.693147 ✓ checked
See also: ReLU and leaky ReLU · sigmoid (logistic function)
\operatorname{softplus}(x)=\ln(1+e^{x})TNR = TN/(TN + FP); NPV = TN/(TN + FN); FPR = 1 − TNR; FNR = 1 − TPR
Worked example: tp = 8, fp = 2, fn = 4, tn = 6 → [0.75, 0.6, 0.25, 0.333333] ✓ checked
\mathrm{TNR}=\frac{TN}{TN+FP},\ \ \mathrm{NPV}=\frac{TN}{TN+FN},\ \ \mathrm{FPR}=\frac{FP}{FP+TN},\ \ \mathrm{FNR}=\frac{FN}{FN+TP}SSIM = (2μxμy + c₁)(2σxy + c₂)/((μx² + μy² + c₁)(σx² + σy² + c₂))
1 for identical images.
Worked example: mx = 0.5, my = 0.5, sx2 = 0.04, sy2 = 0.04, sxy = 0.04, c1 = 0.0001, c2 = 0.0009 → 1 ✓ checked
See also: peak signal-to-noise ratio
\mathrm{SSIM}=\frac{(2\mu_x\mu_y+c_1)(2\sigma_{xy}+c_2)}{(\mu_x^2+\mu_y^2+c_1)(\sigma_x^2+\sigma_y^2+c_2)}SiLU(x) = x σ(x)
Worked example: x = 1 → 0.731059 ✓ checked
See also: GELU · sigmoid (logistic function)
\operatorname{SiLU}(x)=x\,\sigma(x)L = max(0, d(a, p) − d(a, n) + m)
where a = anchor, p = positive, n = negative, m = margin
Pulls matching pairs together and pushes others at least m apart.
Worked example: dp = 0.5, dn = 0.9, m = 0.5 → 0.1 ✓ checked
See also: contrastive loss (Hadsell) · InfoNCE (contrastive) loss
L=\max\big(0,\ d(a,p)-d(a,n)+m\big)T = TP/(TP + αFP + βFN)
α = β = ½ is Dice; β > α favours recall.
Worked example: tp = 8, fp = 2, fn = 4, a = 0.3, b = 0.7 → 0.701754 ✓ checked
See also: Dice loss
T=\frac{TP}{TP+\alpha FP+\beta FN}WER = (S + D + I)/N
where S substitutions, D deletions, I insertions, N reference words
Can exceed 1. CER is the same at character level.
Worked example: S = 2, D = 1, I = 1, N = 20 → 0.2 ✓ checked
See also: Levenshtein distance recurrence
\mathrm{WER}=\frac{S+D+I}{N}J = TPR + TNR − 1; G = √(TPR·TNR)
J picks the best ROC threshold.
Worked example: tpr = 0.8, tnr = 0.7 → [0.5, 0.748331] ✓ checked
See also: area under the ROC curve (AUC)
J=\mathrm{TPR}+\mathrm{TNR}-1,\qquad G=\sqrt{\mathrm{TPR}\cdot\mathrm{TNR}}query the examples the model is least sure about
Worked example: p = [0.5, 0.3, 0.2] → [0.5, 0.2, 1.02965] ✓ checked
See also: epistemic and aleatoric uncertainty (ensembles) · Shannon entropy
\text{least confidence: }1-\max_kp_k;\quad \text{margin: }p_{(1)}-p_{(2)};\quad \text{entropy: }-\sum_kp_k\ln p_kαₜ = ½ ln((1 − εₜ)/εₜ); wᵢ ← wᵢ e^(−αₜ yᵢ hₜ(xᵢ))/Z
Misclassified points are up-weighted.
Worked example: e = 0.2 → 0.693147 ✓ checked
See also: exponential loss · gradient boosting
\alpha_t=\tfrac12\ln\frac{1-\varepsilon_t}{\varepsilon_t},\qquad w_i\leftarrow w_i\,e^{-\alpha_ty_ih_t(x_i)}/Z_tscore = total probability of all classes at least as likely as y
Sets whose size adapts to difficulty; coverage still ≥ 1 − α.
Worked example: p = [0.6, 0.3, 0.1], y = 1 → 0.9 ✓ checked
s(x,y)=\sum_{j:\hat p_j(x)\ge\hat p_y(x)}\hat p_j(x),\qquad \mathcal C(x)=\{y:\ s(x,y)\le\hat q\}uᵢ = (VᵀV + λI)⁻¹Vᵀrᵢ over the items user i rated
Each step is a ridge regression; the basis of many recommender systems.
Worked example: V = [[1, 0], [0, 1], [1, 1]], r = [3, 4, 7], lam = 0 → [3, 4] ✓ checked
See also: ridge regression (closed form)
\mathbf u_i=\Big(V_{\Omega_i}^{\top}V_{\Omega_i}+\lambda I\Big)^{-1}V_{\Omega_i}^{\top}\mathbf r_iθ = arccos(x·y/(‖x‖‖y‖))
Worked example: x = [1, 0], y = [1, 1] → 0.785398 ✓ checked
See also: cosine similarity · dot product
\theta=\arccos\frac{\mathbf x\cdot\mathbf y}{\|\mathbf x\|\|\mathbf y\|}(1 − 1/n)ⁿ → e⁻¹ ≈ 0.368
About 63.2% of the data appears in each bootstrap; the rest is a free validation set (out-of-bag error).
Worked example: n = 100000 → 0.367878 ✓ checked
See also: variance of an averaged ensemble · bootstrap standard error
P(\text{example not in a bootstrap sample})=\Big(1-\frac1n\Big)^n\xrightarrow{n\to\infty}e^{-1}\approx0.368posterior N(m_N, S_N); predictive variance = noise + parameter uncertainty
where α = prior precision, β = noise precision
m_N is the ridge solution with λ = α/β.
Worked example: Phi = [[1, 0], [1, 1], [1, 2]], t = [1, 3, 5], al = 1e-09, be = 1 → [1, 2] ✓ checked
See also: ridge regression (closed form) · Gaussian process posterior
S_N=\big(\alpha I+\beta\Phi^{\top}\Phi\big)^{-1},\ \ \mathbf m_N=\beta S_N\Phi^{\top}\mathbf t,\ \ p(t_*)=\mathcal N\big(\mathbf m_N^{\top}\boldsymbol\phi_*,\ \beta^{-1}+\boldsymbol\phi_*^{\top}S_N\boldsymbol\phi_*\big)idf × f(k₁ + 1)/(f + k₁(1 − b + b|d|/avgdl))
where k₁ ≈ 1.2, b ≈ 0.75
The standard lexical ranking function.
Worked example: N = 1000, df = 10, f = 3, k1 = 1.2, b = 0.75, dl = 100, avg = 100 → 7.14503 ✓ checked
See also: TF-IDF
\mathrm{score}(q,d)=\sum_{t\in q}\ln\frac{N-\mathrm{df}+0.5}{\mathrm{df}+0.5}\cdot\frac{f(k_1+1)}{f+k_1\big(1-b+b\frac{|d|}{\mathrm{avgdl}}\big)}linear, polynomial, Laplace, Matérn 3/2
Worked example: x = [1, 2], y = [3, 1], c = 1, d = 2, l = 1 → [5, 36, 0.106878, 0.10134] ✓ checked
See also: RBF (Gaussian) kernel
k_{\text{lin}}=\mathbf x^{\top}\mathbf x',\quad k_{\text{poly}}=(\mathbf x^{\top}\mathbf x'+c)^d,\quad k_{\text{Laplace}}=e^{-\|\mathbf x-\mathbf x'\|/\ell},\quad k_{\text{Mat\'ern}}^{3/2}=\Big(1+\tfrac{\sqrt3r}{\ell}\Big)e^{-\sqrt3r/\ell}privacy budgets add up; subsampling amplifies privacy
where q = sampling rate
DP-SGD tracks this with the moments (RDP) accountant.
Worked example: e = 1, q = 0.01, k = 100 → [100, 0.0170369] ✓ checked
See also: differential privacy
\text{basic: }\varepsilon_{tot}=\sum_i\varepsilon_i;\qquad \text{advanced: }\varepsilon'=\varepsilon\sqrt{2k\ln(1/\delta')}+k\varepsilon(e^\varepsilon-1);\qquad \text{subsampling: }\ln\big(1+q(e^\varepsilon-1)\big)core point: at least minPts points within ε
No k needed; finds arbitrary shapes and flags noise.
Worked example: d = [0.1, 0.2, 0.9, 0.3], eps = 0.5, minpts = 3 → 1 ✓ checked
See also: k-means objective
N_\varepsilon(p)=\{q:\ d(p,q)\le\varepsilon\};\ \ p\text{ is a core point if }|N_\varepsilon(p)|\ge\text{minPts}ΔDP = |P(ŷ = 1 | A = 0) − P(ŷ = 1 | A = 1)|; disparate impact ≥ 0.8
where A = protected attribute
The four-fifths rule of US employment law.
Worked example: p0 = 0.5, p1 = 0.35 → [0.15, 0.7] ✓ checked
See also: equalised odds and equal opportunity
\Delta_{DP}=\big|P(\hat y=1\mid A=0)-P(\hat y=1\mid A=1)\big|,\qquad \mathrm{DI}=\frac{P(\hat y=1\mid A=1)}{P(\hat y=1\mid A=0)}\ \ (\ge0.8)P(M(D) ∈ S) ≤ e^ε P(M(D′) ∈ S) + δ
Smaller ε means stronger privacy.
Worked example: e = 1 → 2.71828 ✓ checked
See also: Laplace and Gaussian mechanisms
P\big(\mathcal M(D)\in S\big)\le e^{\varepsilon}P\big(\mathcal M(D')\in S\big)+\delta\quad\text{for neighbouring }D,D'R(w) = λ( α‖w‖₁ + (1−α)/2 ‖w‖₂² )
where α mixes L1 and L2
Worked example: w = [1, -2], lam = 0.1, a = 0.5 → 0.275 ✓ checked
See also: L1 penalty (lasso) · L2 penalty (weight decay)
R(\mathbf w)=\lambda\Big(\alpha\|\mathbf w\|_1+\tfrac{1-\alpha}{2}\|\mathbf w\|_2^2\Big)E-step: responsibilities γ; M-step: re-estimate πₖ, μₖ, Σₖ
Each iteration never decreases the likelihood.
Worked example: x = [0, 4], mu = [0, 4], s = [1, 1], pi = [0.5, 0.5] → 0.999665 ✓ checked
See also: Gaussian mixture density · k-means objective
\gamma_{ik}=\frac{\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)}{\sum_j\pi_j\mathcal N(x_i\mid\mu_j,\Sigma_j)};\ \ N_k=\sum_i\gamma_{ik},\ \mu_k=\frac1{N_k}\sum_i\gamma_{ik}x_i,\ \pi_k=\frac{N_k}Nequal TPR and FPR across groups
Several fairness criteria cannot hold simultaneously unless base rates are equal (impossibility theorem).
Worked example: tpr0 = 0.8, tpr1 = 0.7, fpr0 = 0.1, fpr1 = 0.15 → [0.1, 0.05] ✓ checked
See also: demographic parity and disparate impact
P(\hat y=1\mid y=y_0,A=a)\ \text{equal across }a,\ y_0\in\{0,1\};\qquad \text{equal opportunity: only }y=1pairwise interactions through low-rank factors, computed in O(kn)
Worked example: x = [1, 2, 0], V = [[1, 0.5], [0.5, 1], [2, 0]] → [2, 2] ✓ checked
See also: alternating least squares (matrix factorisation)
\hat y=w_0+\sum_iw_ix_i+\sum_{i<j}\langle\mathbf v_i,\mathbf v_j\rangle x_ix_j=w_0+\mathbf w^{\top}\mathbf x+\tfrac12\sum_f\Big[\big(\sum_iv_{if}x_i\big)^2-\sum_iv_{if}^2x_i^2\Big]w ← Σ (n_k/n) w^(k)
Clients train locally; the server averages their weights by data size.
Worked example: n = [100, 300], w = [1, 2] → 1.75 ✓ checked
See also: exponential moving average
w_{t+1}=\sum_{k=1}^{K}\frac{n_k}{n}\,w_{t+1}^{(k)}w ∝ S_W⁻¹(μ₁ − μ₀)
where S_W = within-class scatter
Worked example: Sw = [[2, 0], [0, 1]], d = [2, 1] → [1, 1] ✓ checked
See also: Mahalanobis distance
\mathbf w\propto S_W^{-1}(\boldsymbol\mu_1-\boldsymbol\mu_0),\qquad J(\mathbf w)=\frac{\big(\mathbf w^{\top}(\boldsymbol\mu_1-\boldsymbol\mu_0)\big)^2}{\mathbf w^{\top}S_W\mathbf w}δₖ(x) = −½(x − μₖ)ᵀΣₖ⁻¹(x − μₖ) − ½ ln|Σₖ| + ln πₖ
where shared Σ gives LDA (linear boundary), per-class Σₖ gives QDA
Worked example: x = [1], mu = 0, S = 1, pi = 0.5 → -1.19315 ✓ checked
See also: Fisher's linear discriminant
\delta_k(x)=-\tfrac12(x-\mu_k)^{\top}\Sigma_k^{-1}(x-\mu_k)-\tfrac12\ln|\Sigma_k|+\ln\pi_kp(x) = Σₖ πₖ N(x | μₖ, σₖ²)
where πₖ = mixing weights
Fit by expectation–maximisation.
Worked example: pi = [0.5, 0.5], mu = [0, 2], s = [1, 1], x = 0 → 0.226467 ✓ checked
See also: normal (Gaussian) density
p(x)=\sum_{k}\pi_k\,\mathcal N(x\mid\mu_k,\sigma_k^2),\quad \sum_k\pi_k=1log prior + Σ per-feature Gaussian log-likelihoods
Worked example: x = [1, 2], mu = [0, 2], s2 = [1, 4], prior = 0.5 → -3.72417 ✓ checked
See also: naive Bayes
\ln P(y\mid\mathbf x)=\ln P(y)-\sum_i\Big[\tfrac12\ln(2\pi\sigma_{iy}^2)+\frac{(x_i-\mu_{iy})^2}{2\sigma_{iy}^2}\Big]+\text{const}ln p(y) = −½yᵀA⁻¹y − ½ ln|A| − (n/2) ln 2π, A = K + σ²I
Maximise over kernel hyperparameters: a built-in Occam's razor.
Worked example: K = [[1, 0.5], [0.5, 1]], y = [1, 2], s2 = 0.1 → -3.64038 ✓ checked
See also: Gaussian process posterior
\ln p(\mathbf y)=-\tfrac12\mathbf y^{\top}(K+\sigma^2I)^{-1}\mathbf y-\tfrac12\ln|K+\sigma^2I|-\tfrac n2\ln2\piμ* = K*ᵀ(K + σ²I)⁻¹y; σ*² = k** − K*ᵀ(K + σ²I)⁻¹K*
Exact Bayesian regression with a kernel prior; O(n³).
Worked example: K = [[1, 0.5], [0.5, 1]], ks = [0.8, 0.6], kss = 1, y = [1, 2], s2 = 0.1 → [1.14583, 0.354167] ✓ checked
See also: multivariate normal: conditional distribution · kernel ridge regression
\mu_*=K_*^{\top}(K+\sigma^2I)^{-1}\mathbf y,\qquad \sigma_*^2=k_{**}-K_*^{\top}(K+\sigma^2I)^{-1}K_*G = 1 − Σ pₖ²
where pₖ = class proportions in a node
0 for a pure node.
Worked example: p = [0.5, 0.5] → 0.5 ✓ checked
See also: information gain
G=1-\sum_k p_k^2J = Σ f(Xᵢⱼ)(wᵢᵀw̃ⱼ + bᵢ + b̃ⱼ − ln Xᵢⱼ)²
Worked example: x = 10, xmax = 100, s = 2 → 0.0162815 ✓ checked
See also: skip-gram with negative sampling
J=\sum_{i,j}f(X_{ij})\big(\mathbf w_i^{\top}\tilde{\mathbf w}_j+b_i+\tilde b_j-\ln X_{ij}\big)^2,\quad f(x)=\min\big((x/x_{\max})^{3/4},1\big)fit each tree to the negative gradient of the loss
where ν = shrinkage
For squared error the pseudo-residual is y − F; for log loss it is y − p.
Worked example: y = 1, F = 0 → [1, 0.5] ✓ checked
See also: XGBoost leaf weight and split gain · AdaBoost
F_m(x)=F_{m-1}(x)+\nu\,h_m(x),\quad h_m\ \text{fits}\ r_{im}=-\Big[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\Big]_{F=F_{m-1}}∇_w L = (p − y)·x; ∂L/∂b = p − y
where p = σ(wᵀx + b)
Residual times input.
Worked example: p = 0.8, y = 1, x = [1, 2] → [-0.2, -0.4] ✓ checked
See also: binary cross-entropy (log loss) · gradient of softmax cross-entropy
\nabla_{\mathbf w}L=(p-y)\,\mathbf x,\qquad \frac{\partial L}{\partial b}=p-y∂L/∂zₖ = pₖ − yₖ
where p = softmax(z), y = one-hot target
The simplest gradient in deep learning.
Worked example: z = [0, 0], y = [1, 0] → [-0.5, 0.5] ✓ checked
See also: softmax cross-entropy (from logits) · gradient of logistic loss
\frac{\partial L}{\partial z_k}=p_k-y_kreplace the score by the empirical positive rate of its bin
Worked example: pos = 18, n = 30 → 0.6 ✓ checked
See also: expected calibration error (ECE) · Platt scaling
\hat p(s)=\frac{\#\{i\in\text{bin}(s):y_i=1\}}{\#\{i\in\text{bin}(s)\}}αₜ(j) = [Σᵢ αₜ₋₁(i)Aᵢⱼ] Bⱼ(oₜ)
where A = transitions, B = emissions
Dynamic programming in O(TN²). Replace the sum by a max for Viterbi.
Worked example: A = [[0.7, 0.3], [0.4, 0.6]], B = [[0.9, 0.1], [0.2, 0.8]], pi = [0.6, 0.4], o = [0, 1] → 0.209 ✓ checked
See also: Viterbi algorithm
\alpha_t(j)=\Big[\sum_i\alpha_{t-1}(i)A_{ij}\Big]B_j(o_t),\quad P(O)=\sum_j\alpha_T(j)IG = H(parent) − Σⱼ (nⱼ/n) H(childⱼ)
The split criterion of ID3/C4.5 trees.
Worked example: hp = 0.693147, w = [0.5, 0.5], hc = [0, 0] → 0.693147 ✓ checked
See also: Shannon entropy · Gini impurity
\mathrm{IG}=H(\text{parent})-\sum_j\frac{n_j}{n}H(\text{child}_j)pool adjacent violators: replace out-of-order neighbours by their mean
A monotone, non-parametric fit; used to calibrate scores.
Worked example: y = [1, 3, 2, 4] → [1, 2.5, 2.5, 4] ✓ checked
See also: Platt scaling
\min\sum(y_i-f_i)^2\ \text{s.t. }f_1\le f_2\le\cdots\le f_nJ = Σₙ Σₖ rₙₖ ‖xₙ − μₖ‖²
where rₙₖ = 1 if point n is assigned to cluster k
Lloyd's algorithm alternates assignment and mean updates; J never increases.
Worked example: X = [[0, 0], [2, 0]], mu = [1, 0] → 2 ✓ checked
See also: principal component analysis
J=\sum_{n}\sum_{k}r_{nk}\,\|\mathbf x_n-\boldsymbol\mu_k\|^2assign to the nearest centroid, then move each centroid to its cluster mean
Worked example: x = [[0, 0], [0, 2], [10, 0]], mu = [[0, 1], [10, 0]] → [[0, 1], [10, 0]] ✓ checked
See also: k-means objective · k-means++ seeding
c_i=\arg\min_k\|x_i-\mu_k\|^2,\qquad \mu_k=\frac1{|C_k|}\sum_{i\in C_k}x_ichoose the next centre with probability ∝ D(x)²
where D(x) = distance to the nearest chosen centre
O(log k)-competitive with the optimum in expectation.
Worked example: d = [0, 1, 3] → [0, 0.1, 0.9] ✓ checked
See also: k-means update steps
P(x)=\frac{D(x)^2}{\sum_{x'}D(x')^2}majority (or distance-weighted) vote of the k nearest points
Worked example: labels = [1, 1, 0], d = [1, 2, 0.5] → 0 ✓ checked
See also: Cover–Hart bound for nearest neighbours
\hat y(x)=\arg\max_c\sum_{i\in N_k(x)}w_i\mathbf 1[y_i=c],\quad w_i=\frac1{d(x,x_i)}\ \text{(optional)}predict with F, Q; update with gain K = P⁻Hᵀ(HP⁻Hᵀ + R)⁻¹
The exact Bayesian filter for linear-Gaussian state-space models.
Worked example: x = 0, P = 1, z = 2, R = 1 → [1, 0.5] ✓ checked
See also: multivariate normal: conditional distribution · HMM forward algorithm
\text{predict: }\hat x^-=F\hat x,\ P^-=FPF^{\top}+Q;\quad \text{update: }K=P^-H^{\top}(HP^-H^{\top}+R)^{-1},\ \hat x=\hat x^-+K(z-H\hat x^-),\ P=(I-KH)P^-K̃ = K − 1ₙK − K1ₙ + 1ₙK1ₙ
Centres the data in feature space; then eigendecompose K̃.
Worked example: K = [[2, 1], [1, 2]] → [[0.5, -0.5], [-0.5, 0.5]] ✓ checked
See also: PCA via SVD
\tilde K=K-\mathbf 1_nK-K\mathbf 1_n+\mathbf 1_nK\mathbf 1_n,\quad \mathbf 1_n=\tfrac1n\mathbf 1\mathbf 1^{\top}α = (K + λI)⁻¹y; f(x) = Σ αᵢ k(xᵢ, x)
Equals the GP posterior mean with noise variance λ.
Worked example: K = [[1, 0.5], [0.5, 1]], y = [1, 2], lam = 0.5 → [0.25, 1.25] ✓ checked
See also: Gaussian process posterior · RBF (Gaussian) kernel
\boldsymbol\alpha=(K+\lambda I)^{-1}\mathbf y,\qquad \hat f(x)=\sum_i\alpha_i\,k(x_i,x)R(w) = λ Σ |wᵢ|
where λ = strength
Drives weights exactly to zero (sparse solutions).
Worked example: w = [1, -2], lam = 0.1 → 0.3 ✓ checked
See also: L1 norm (Manhattan) · L2 penalty (weight decay) · elastic net
R(\mathbf w)=\lambda\|\mathbf w\|_1R(w) = λ Σ wᵢ²
Shrinks weights smoothly. Equivalent to a Gaussian prior on w.
Worked example: w = [1, -2], lam = 0.1 → 0.5 ✓ checked
See also: L1 penalty (lasso) · ridge regression (closed form)
R(\mathbf w)=\lambda\|\mathbf w\|_2^2add Laplace noise of scale Δf/ε
where Δf = sensitivity
Worked example: d = 1, e = 0.5, dl = 1e-05 → [2, 9.68961] ✓ checked
See also: differential privacy
\mathcal M(D)=f(D)+\operatorname{Lap}\big(\Delta f/\varepsilon\big),\qquad \sigma=\frac{\Delta f\sqrt{2\ln(1.25/\delta)}}\varepsilon\ \ (\text{Gaussian})ŷ = wᵀx + b
where w = weights, b = bias
Worked example: w = [1, 2], x = [3, 4], b = 0.5 → 11.5 ✓ checked
See also: ordinary least squares (normal equations) · logistic regression model
\hat y=\mathbf w^{\top}\mathbf x+bp(y=1|x) = σ(wᵀx + b)
The log-odds are linear in x.
Worked example: w = [1, -1], x = [2, 2], b = 0 → 0.5 ✓ checked
See also: sigmoid (logistic function) · binary cross-entropy (log loss) · gradient of logistic loss
p(y=1\mid\mathbf x)=\sigma(\mathbf w^{\top}\mathbf x+b)P(y|x) ∝ P(y) Πᵢ P(xᵢ|y)
where Features assumed independent given the class
In practice add log-probabilities.
Worked example: prior = 0.5, lik = [0.8, 0.5] → 0.2 ✓ checked
See also: Bayes' theorem
P(y\mid\mathbf x)\propto P(y)\prod_i P(x_i\mid y)H ← H ∘ (WᵀV)/(WᵀWH); W ← W ∘ (VHᵀ)/(WHHᵀ)
Lee–Seung updates; keep everything non-negative and never increase ‖V − WH‖.
Worked example: V = [[1, 2], [2, 4]], W = [[1], [1]], H = [[1, 1]] → [[1.5, 3]] ✓ checked
See also: PCA via SVD
H\leftarrow H\circ\frac{W^{\top}V}{W^{\top}WH},\qquad W\leftarrow W\circ\frac{VH^{\top}}{WHH^{\top}}wᵢ ∝ wᵢ p(z | xᵢ); N_eff = 1/Σwᵢ²
Sequential Monte Carlo for non-linear, non-Gaussian filtering.
Worked example: w = [0.5, 0.3, 0.2] → 2.63158 ✓ checked
See also: effective sample size · Kalman filter
w_i\propto w_i\,p(z\mid x_i),\quad N_{\text{eff}}=\frac1{\sum_iw_i^2}\ \text{(resample if small)}components = V, variances = σₖ²/(n − 1)
where X_c = centred data
Worked example: X = [[1, 0], [-1, 0], [0, 2], [0, -2]] → [2.66667, 0.666667] ✓ checked
See also: principal component analysis · singular value decomposition
X_c=U\Sigma V^{\top},\quad \text{components }=V,\quad \text{scores }=U\Sigma,\quad \text{variance}_k=\frac{\sigma_k^2}{n-1}mistakes ≤ (R/γ)²
where R = max ‖x‖, γ = margin
Worked example: R = 2, g = 0.5 → 16 ✓ checked
See also: perceptron update
\#\text{mistakes}\le\Big(\frac R\gamma\Big)^2if a mistake: w ← w + y x
Converges in at most (R/γ)² mistakes when the data are separable with margin γ.
Worked example: w = [0, 0], x = [1, 2], y = 1 → [1, 2] ✓ checked
See also: support vector machine (soft margin)
\text{if }y\,\mathbf w^{\top}\mathbf x\le0:\quad \mathbf w\leftarrow\mathbf w+y\,\mathbf xP(y = 1 | s) = σ(As + B)
where s = classifier score; A, B fitted on validation data by maximum likelihood
Calibrates SVM and boosting scores; temperature scaling is the one-parameter version.
Worked example: A = 2, B = -1, s = 1 → 0.731059 ✓ checked
See also: isotonic regression (pool adjacent violators)
P(y=1\mid s)=\sigma(As+B)σ̂² = mean of the discarded eigenvalues
PCA as a latent-variable model; gives a likelihood.
Worked example: lam = [4, 1, 0.5, 0.5], q = 2 → 0.5 ✓ checked
See also: principal component analysis
\hat\sigma^2=\frac1{d-q}\sum_{j=q+1}^{d}\lambda_j,\qquad W=U_q(\Lambda_q-\hat\sigma^2I)^{1/2}answer truthfully with probability e^ε/(1 + e^ε), and debias the aggregate
The simplest local-DP mechanism.
Worked example: e = 1, pt = 0.4 → 0.4 ✓ checked
See also: differential privacy
P(\text{truth})=\frac{e^\varepsilon}{1+e^\varepsilon},\qquad \hat p=\frac{\tilde p-(1-q)}{2q-1}k(x, x′) = exp(−‖x − x′‖²/2ℓ²)
where ℓ = length scale, γ = 1/(2ℓ²)
Positive definite; corresponds to an infinite-dimensional feature space.
Worked example: x = [0, 0], y = [1, 1], l = 1 → 0.367879 ✓ checked
See also: Gram matrix · Gaussian process posterior
k(\mathbf x,\mathbf x')=\exp\!\Big(-\frac{\|\mathbf x-\mathbf x'\|^2}{2\ell^2}\Big)=\exp\!\big(-\gamma\|\mathbf x-\mathbf x'\|^2\big)L = −ln σ(u_oᵀv_c) − Σ ln σ(−u_kᵀv_c)
where o = observed context word, k = sampled negatives
Implicitly factorises the shifted PMI matrix.
Worked example: pos = 2, neg = [-1, 0.5] → 1.41427 ✓ checked
See also: sigmoid (logistic function) · GloVe loss
L=-\ln\sigma(\mathbf u_o^{\top}\mathbf v_c)-\sum_{k=1}^{K}\ln\sigma\big(-\mathbf u_k^{\top}\mathbf v_c\big)min ½‖w‖² + C Σ ξᵢ s.t. yᵢ(wᵀxᵢ + b) ≥ 1 − ξᵢ, ξᵢ ≥ 0
where C trades margin width against violations ξ
Worked example: w = [3, 4], C = 1, xi = [0, 0.5] → 13 ✓ checked
See also: SVM margin width · hinge loss · KKT stationarity (inequality constraints)
\min_{\mathbf w,b,\xi}\ \tfrac12\|\mathbf w\|^2+C\sum_i\xi_i\quad\text{s.t. }y_i(\mathbf w^{\top}\mathbf x_i+b)\ge1-\xi_i,\ \xi_i\ge0margin = 2 / ‖w‖
Maximising the margin means minimising ‖w‖.
Worked example: w = [3, 4] → 0.4 ✓ checked
See also: support vector machine (soft margin)
\text{margin}=\frac{2}{\|\mathbf w\|}Gaussian affinities in the data space, Student-t (1 dof) affinities in the map, minimise KL
The heavy-tailed t kernel avoids crowding. Perplexity = 2^H(Pᵢ) sets σᵢ.
Worked example: d2 = 1 → 0.5 ✓ checked
See also: Kullback–Leibler divergence
p_{j|i}=\frac{e^{-\|x_i-x_j\|^2/2\sigma_i^2}}{\sum_{k\ne i}e^{-\|x_i-x_k\|^2/2\sigma_i^2}},\quad q_{ij}=\frac{(1+\|y_i-y_j\|^2)^{-1}}{\sum_{k\ne l}(1+\|y_k-y_l\|^2)^{-1}},\quad C=\mathrm{KL}(P\|Q)tf-idf = tf × ln(N/df)
where N = documents, df = documents containing t
Common-word terms get weights near 0.
Worked example: tf = 3, N = 1000, df = 10 → 13.8155 ✓ checked
See also: BM25
\mathrm{tfidf}(t,d)=\mathrm{tf}(t,d)\cdot\ln\frac{N}{\mathrm{df}(t)}Var = ρσ² + (1−ρ)σ²/B
where B models, pairwise correlation ρ, variance σ²
Averaging only removes the uncorrelated part: decorrelate the models (random forests).
Worked example: s2 = 1, rho = 0.5, B = 10 → 0.55 ✓ checked
See also: bias–variance decomposition
\operatorname{Var}\!\Big(\frac1B\sum_b f_b\Big)=\rho\sigma^2+\frac{1-\rho}{B}\sigma^2δₜ(j) = maxᵢ[δₜ₋₁(i)Aᵢⱼ] Bⱼ(oₜ)
The most likely hidden-state sequence.
Worked example: A = [[0.7, 0.3], [0.4, 0.6]], B = [[0.9, 0.1], [0.2, 0.8]], pi = [0.6, 0.4], o = [0, 1] → 0.1296 ✓ checked
See also: HMM forward algorithm
\delta_t(j)=\max_i\big[\delta_{t-1}(i)A_{ij}\big]B_j(o_t),\quad \text{backtrack the argmax}Δ = |A||B|/(|A| + |B|) · ‖μ_A − μ_B‖²
Merges the pair that least increases the within-cluster variance.
Worked example: nA = 3, nB = 5, d2 = 4 → 7.5 ✓ checked
See also: k-means objective
\Delta(A,B)=\frac{|A||B|}{|A|+|B|}\,\|\mu_A-\mu_B\|^2w* = −G/(H + λ); gain = ½[G_L²/(H_L + λ) + G_R²/(H_R + λ) − (G_L + G_R)²/(H + λ)] − γ
where G, H = sums of first and second derivatives of the loss in the leaf
Worked example: GL = -4, HL = 4, GR = 3, HR = 3, lam = 1, g = 0 → [0.8, 2.6625] ✓ checked
See also: gradient boosting
w^*=-\frac{G}{H+\lambda},\qquad \text{Gain}=\tfrac12\Big[\frac{G_L^2}{H_L+\lambda}+\frac{G_R^2}{H_R+\lambda}-\frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\Big]-\gammaeᵢⱼ = vᵀ tanh(W₁sᵢ + W₂hⱼ); α = softmax(e)
The original neural-translation attention.
Worked example: e = [1, 2, 3] → [0.0900306, 0.244728, 0.665241] ✓ checked
See also: scaled dot-product attention
e_{ij}=\mathbf v^{\top}\tanh(W_1\mathbf s_i+W_2\mathbf h_j),\quad \alpha_{ij}=\operatorname{softmax}_j(e_{ij}),\quad \mathbf c_i=\sum_j\alpha_{ij}\mathbf h_jq = round(x/s) + z; x̂ = s(q − z)
where b bits, scale s, zero-point z
Worst-case rounding error is s/2.
Worked example: lo = -1, hi = 1, b = 8, x = 0.3 → 1 ✓ checked
See also: machine epsilon and floating-point precision
q=\operatorname{clip}\big(\operatorname{round}(x/s)+z,\ 0,\ 2^b-1\big),\quad s=\frac{x_{\max}-x_{\min}}{2^b-1},\quad \hat x=s\,(q-z)score − m·(i − j)
where m = head-specific slope
A linear distance penalty instead of position embeddings; extrapolates to longer contexts.
Worked example: qk = 2, m = 0.5, i = 10, j = 6 → 0 ✓ checked
See also: rotary position embedding (RoPE)
\operatorname{softmax}\big(q_i k_j^{\top}-m\,(i-j)\big),\quad m_h=2^{-8h/H}δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾)ᵀ δ⁽ˡ⁺¹⁾ ⊙ σ′(z⁽ˡ⁾)
∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾ (a⁽ˡ⁻¹⁾)ᵀ
Worked example: W = [[1, 2]], d = [0.5], sp = [1, 0.5] → [0.5, 0.5] ✓ checked
See also: chain rule
\boldsymbol\delta^{(l)}=\big(W^{(l+1)}\big)^{\top}\boldsymbol\delta^{(l+1)}\odot\sigma'\!\big(\mathbf z^{(l)}\big)x̂ = (x − μ_B)/√(σ²_B + ε); y = γx̂ + β
where μ_B, σ²_B = batch statistics; γ, β learned
Uses running averages at inference time.
Worked example: x = [1, 3], eps = 1e-05 → [-0.999995, 0.999995] ✓ checked
See also: layer normalisation · RMSNorm
\hat x=\frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}},\quad y=\gamma\hat x+\betarunning = (1 − m)·running + m·batch (m ≈ 0.1)
Used at inference time in place of batch statistics.
Worked example: run = 0, b = 1, m = 0.1 → 0.1 ✓ checked
See also: batch normalisation · exponential moving average
\mu_{\text{run}}\leftarrow(1-m)\mu_{\text{run}}+m\,\mu_B,\qquad \sigma^2_{\text{run}}\leftarrow(1-m)\sigma^2_{\text{run}}+m\,\sigma^2_Bscore = ln P(Y|X) / ((5 + |Y|)/6)^α
where α ≈ 0.6
Without it beam search prefers short outputs.
Worked example: lp = -6, n = 7, a = 0.6 → -3.95852 ✓ checked
See also: top-k, top-p and temperature sampling
\operatorname{score}(Y)=\frac{\ln P(Y\mid X)}{\big((5+|Y|)/6\big)^{\alpha}}L = −ln σ(r(x, y_w) − r(x, y_l))
where y_w preferred over y_l
Trains a reward model from human preference pairs.
Worked example: rw = 1, rl = 0.2 → 0.371101 ✓ checked
See also: sigmoid (logistic function) · binary cross-entropy (log loss)
L=-\ln\sigma\big(r(x,y_w)-r(x,y_l)\big)ε̃ = (1 + w)ε(x, c) − wε(x, ∅)
where w = guidance strength
Extrapolates away from the unconditional prediction toward the conditioned one.
Worked example: ec = 0.8, eu = 0.2, w = 3 → 2.6 ✓ checked
See also: DDPM reverse step
\tilde\varepsilon=(1+w)\,\varepsilon_\theta(x_t,c)-w\,\varepsilon_\theta(x_t,\varnothing)D ≈ 20N tokens; N and D both grow as √C
Scale parameters and data together. Many deployed models are deliberately over-trained for cheaper inference.
Worked example: C = 5.76e+23 → [6.9282e+10, 1.38564e+12] ✓ checked
See also: parametric scaling law · training compute estimate
D^*\approx20\,N,\qquad N^*\propto C^{0.5},\ D^*\propto C^{0.5},\qquad C\approx6NDFLOPs ≈ 2·H_out·W_out·k²·C_in·C_out
Worked example: H = 56, W = 56, k = 3, ci = 64, co = 64 → 2.31211e+08 ✓ checked
See also: depthwise-separable convolution parameters · matrix-multiplication cost
\text{FLOPs}\approx2\,H_{out}W_{out}\,k^2\,C_{in}C_{out},\qquad \text{params}=k^2C_{in}C_{out}+C_{out}n_out = ⌊(n + 2p − k)/s⌋ + 1
where n input size, k kernel, p padding, s stride
Worked example: n = 224, k = 7, p = 3, s = 2 → 112 ✓ checked
See also: parameter counts
n_{out}=\left\lfloor\frac{n+2p-k}{s}\right\rfloor+1predict x̂₀, then jump to the earlier noise level
Allows far fewer sampling steps than DDPM.
Worked example: x = 1, e = 0.5, ab_t = 0.5, ab_p = 0.8 → 1.0413 ✓ checked
See also: DDPM reverse step
\hat x_0=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta}{\sqrt{\bar\alpha_t}},\qquad x_{t-1}=\sqrt{\bar\alpha_{t-1}}\,\hat x_0+\sqrt{1-\bar\alpha_{t-1}}\,\varepsilon_\thetaμ = (xₜ − βₜ/√(1 − ᾱₜ) ε_θ)/√αₜ; β̃ₜ = (1 − ᾱₜ₋₁)/(1 − ᾱₜ) βₜ
Worked example: x = 1, e = 0.5, beta = 0.02, ab = 0.5 → 0.995867 ✓ checked
See also: diffusion forward process · diffusion training loss
\mu_\theta(x_t,t)=\frac1{\sqrt{\alpha_t}}\Big(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon_\theta(x_t,t)\Big),\qquad \tilde\beta_t=\frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_ttrain s_θ(x̃) to equal −(x̃ − x)/σ² = ∇log q(x̃|x)
The score is ∇ₓ log p(x); ε-prediction in DDPM is the same thing up to scale: s = −ε/σ.
Worked example: x = 0, xt = 0.5, s = 0.5 → -2 ✓ checked
See also: diffusion training loss · Langevin dynamics
\mathcal L=\mathbb E\Big\|s_\theta(\tilde x,\sigma)+\frac{\tilde x-x}{\sigma^2}\Big\|^2,\qquad \tilde x=x+\sigma\varepsilonk²Cᵢₙ + CᵢₙC_out vs k²CᵢₙC_out
About an 8–9× reduction for 3×3 kernels (MobileNet).
Worked example: k = 3, ci = 64, co = 128 → [8768, 73728] ✓ checked
See also: parameter counts · convolution cost
\underbrace{k^2C_{in}+C_{in}C_{out}}_{\text{separable}}\ \text{vs}\ \underbrace{k^2C_{in}C_{out}}_{\text{standard}}xₜ = √ᾱₜ x₀ + √(1−ᾱₜ) ε; ᾱₜ = Πₛ αₛ
where ε ~ N(0, I)
Sample any noise level directly from x₀.
Worked example: alphas = [0.9, 0.8], x0 = 1, e = 1 → 1.37768 ✓ checked
See also: diffusion training loss
x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\quad \bar\alpha_t=\prod_{s\le t}\alpha_sL = E‖ε − ε_θ(xₜ, t)‖²
The network predicts the noise that was added.
Worked example: e = [1, 0], ep = [0.5, 0] → 0.25 ✓ checked
See also: diffusion forward process
L=\mathbb{E}_{t,x_0,\varepsilon}\big[\|\varepsilon-\varepsilon_\theta(x_t,t)\|^2\big]k_eff = k + (k − 1)(d − 1)
where d = dilation
Grows the receptive field without adding parameters.
Worked example: k = 3, d = 4 → 9 ✓ checked
See also: receptive field recurrence
k_{\text{eff}}=k+(k-1)(d-1)L = −ln σ(β[log(π/π_ref)(y_w) − log(π/π_ref)(y_l)])
Optimises the RLHF objective in closed form, with no separate reward model or RL loop.
Worked example: b = 0.1, lw = -10, lw_ref = -12, ll = -15, ll_ref = -14 → 0.554355 ✓ checked
See also: Bradley–Terry reward-model loss · RLHF reward with KL penalty
L=-\ln\sigma\Big(\beta\ln\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\ln\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big)stop after `patience` epochs without a new best validation loss
Acts like an L2 penalty whose strength shrinks with training time.
Worked example: hist = [0.9, 0.8, 0.82, 0.81, 0.83], p = 3 → 1 ✓ checked
See also: L2 penalty (weight decay)
\text{stop when }L_{\text{val}}\text{ has not improved for }p\text{ epochs (patience)}ELU: x for x > 0, α(eˣ − 1) otherwise; SELU scales it by λ
SELU self-normalises activations in fully connected networks.
Worked example: x = -1, a = 1 → -0.632121 ✓ checked
See also: ReLU and leaky ReLU · GELU
\operatorname{ELU}(x)=\begin{cases}x&x>0\\ \alpha(e^x-1)&x\le0\end{cases},\qquad \operatorname{SELU}=\lambda\operatorname{ELU}\ (\lambda\approx1.0507,\ \alpha\approx1.6733)β = 0.5^(1/half-life); window ≈ 1/(1 − β)
Worked example: b = 0.99 → 100 ✓ checked
See also: exponential moving average
\beta=0.5^{1/\text{half-life}},\qquad \text{effective window}\approx\frac1{1-\beta}log p(x) ≥ E_q[log p(x|z)] − KL(q(z|x) ‖ p(z))
Reconstruction term minus a regulariser.
Worked example: recon = -10, kl = 2 → -12 ✓ checked
See also: VAE KL term (diagonal Gaussian) · reparameterisation trick
\log p(x)\ \ge\ \mathbb{E}_{q(z|x)}[\log p(x\mid z)]-D_{\mathrm{KL}}\big(q(z\mid x)\,\|\,p(z)\big)x_t = (1 − t)x₀ + t x₁; regress v_θ onto x₁ − x₀
where x₀ ~ noise, x₁ ~ data
Straight-line paths give simple training and few-step sampling.
Worked example: x0 = 0, x1 = 2, t = 0.25 → [0.5, 2] ✓ checked
See also: diffusion forward process
x_t=(1-t)\,x_0+t\,x_1,\qquad \mathcal L=\mathbb E\big\|v_\theta(x_t,t)-(x_1-x_0)\big\|^2min_G max_D E[log D(x)] + E[log(1 − D(G(z)))]
Worked example: dx = 0.8, dg = 0.3 → -0.579818 ✓ checked
See also: optimal GAN discriminator
\min_G\max_D\ \mathbb{E}_{x\sim p_{\text{data}}}[\log D(x)]+\mathbb{E}_{z\sim p_z}[\log(1-D(G(z)))]GLU(a, b) = a ⊙ σ(b)
Worked example: a = 2, b = 0 → 1 ✓ checked
See also: SwiGLU feed-forward block · sigmoid (logistic function)
\operatorname{GLU}(a,b)=a\odot\sigma(b)GELU ≈ ½x(1 + tanh(√(2/π)(x + 0.044715x³)))
The form used in the original BERT and GPT-2 code.
Worked example: x = 1 → 0.841192 ✓ checked
See also: GELU
\operatorname{GELU}(x)\approx\tfrac12x\Big(1+\tanh\!\Big[\sqrt{\tfrac2\pi}\big(x+0.044715x^3\big)\Big]\Big)R ≤ R̂ + √((ln|H| + ln(2/δ))/(2n))
From Hoeffding and a union bound.
Worked example: H = 1e+06, d = 0.05, n = 10000 → 0.0295841 ✓ checked
See also: Hoeffding's inequality · VC generalisation bound
R(h)\le\hat R(h)+\sqrt{\frac{\ln|\mathcal H|+\ln(2/\delta)}{2n}}\quad\text{with probability }1-\deltalearned attention weights over neighbours
Worked example: e = [0, 0] → [0.5, 0.5] ✓ checked
See also: scaled dot-product attention · graph convolution (GCN)
e_{ij}=\operatorname{LeakyReLU}\big(\mathbf a^{\top}[W\mathbf h_i\,\|\,W\mathbf h_j]\big),\quad \alpha_{ij}=\operatorname{softmax}_j(e_{ij}),\quad \mathbf h_i'=\sigma\Big(\sum_j\alpha_{ij}W\mathbf h_j\Big)H′ = σ(D̃^(−½) Ã D̃^(−½) H W)
Each node averages its neighbours (and itself).
Worked example: A = [[0, 1], [1, 0]], H = [[1], [3]] → [[2], [2]] ✓ checked
See also: normalised graph Laplacian
H^{(l+1)}=\sigma\big(\tilde D^{-1/2}\tilde A\tilde D^{-1/2}H^{(l)}W^{(l)}\big),\quad \tilde A=A+Ih′ = MLP((1 + ε)hᵥ + Σ hᵤ)
As expressive as the Weisfeiler–Lehman test.
Worked example: h = 1, nb = [2, 3], e = 0.1 → 6.1 ✓ checked
See also: graph convolution (GCN)
\mathbf h_v'=\operatorname{MLP}\Big((1+\varepsilon)\mathbf h_v+\sum_{u\in N(v)}\mathbf h_u\Big)normalise over each group of channels
Independent of batch size; instance norm is G = C, layer norm is G = 1.
Worked example: x = [1, 3, 5, 7], eps = 1e-12 → [-1.34164, -0.447214, 0.447214, 1.34164] ✓ checked
See also: layer normalisation · batch normalisation
\hat x=\frac{x-\mu_g}{\sqrt{\sigma_g^2+\epsilon}},\quad \mu_g,\sigma_g\ \text{over the channels in a group and }H,WÂᵢ = (rᵢ − mean(r))/std(r) over a group of G samples
Replaces a learned value function by the group baseline.
Worked example: r = [1, 0, 0, 1] → [1, -1, -1, 1] ✓ checked
See also: generalised advantage estimation · PPO clipped objective
\hat A_i=\frac{r_i-\operatorname{mean}(r_{1:G})}{\operatorname{std}(r_{1:G})}k² C_in C_out / g
where g = number of groups
g = C_in gives a depthwise convolution.
Worked example: k = 3, ci = 64, co = 64, g = 8 → 4608 ✓ checked
See also: depthwise-separable convolution parameters
\frac{k^2C_{in}C_{out}}{g}hₜ = (1 − zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ
where z = update gate
Worked example: z = 0.25, hp = 4, ht = 0 → 3 ✓ checked
See also: LSTM cell state update
h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_tstd = √(2 / fan_in)
For ReLU networks.
Worked example: fan_in = 50 → 0.2 ✓ checked
See also: Xavier (Glorot) initialisation
w\sim\mathcal N\!\left(0,\tfrac{2}{n_{in}}\right)L_K = E log (1/K) Σ p(x, zₖ)/q(zₖ|x)
A tighter bound as K grows.
Worked example: w = [0.5, 1.5] → 0 ✓ checked
See also: evidence lower bound (VAE) · importance sampling
\mathcal L_K=\mathbb E\Big[\log\frac1K\sum_{k=1}^{K}\frac{p(x,z_k)}{q(z_k\mid x)}\Big]\ \ge\ \mathcal L_1=\text{ELBO}L = −log( exp(s⁺/τ) / Σⱼ exp(sⱼ/τ) )
where s = similarity scores (positive first), τ = temperature
Lower bounds mutual information.
Worked example: s = [1, 0], tau = 1 → 0.313262 ✓ checked
See also: cosine similarity · mutual information
L=-\log\frac{\exp(s^{+}/\tau)}{\sum_{j}\exp(s_j/\tau)}y = (m ⊙ x) / (1 − p)
where p = drop probability
Scaling by 1/(1−p) keeps the expected activation unchanged.
Worked example: m = [1, 0], x = [2, 2], p = 0.5 → [4, 0] ✓ checked
See also: L2 penalty (weight decay)
\mathbf y=\frac{\mathbf m\odot\mathbf x}{1-p},\quad m_i\sim\mathrm{Bernoulli}(1-p)bytes = 2 · layers · kv_heads · d_head · T · bytes_per_value
Grouped-query attention shrinks it by n_heads/n_kv. It, not weights, often limits long-context serving.
Worked example: L = 32, nkv = 8, dh = 128, T = 8192, b = 2 → 1.07374e+09 ✓ checked
See also: self-attention cost
\text{bytes}=2\cdot n_{\text{layers}}\cdot n_{kv}\cdot d_{\text{head}}\cdot T\cdot bx̂ᵢ = (xᵢ − μ)/√(σ² + ε); yᵢ = γᵢx̂ᵢ + βᵢ
where μ, σ² over the features of one example
Independent of batch size; standard in transformers.
Worked example: x = [1, 2, 3], eps = 1e-05 → [-1.22474, 0, 1.22474] ✓ checked
See also: batch normalisation · RMSNorm
\hat x_i=\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}},\quad y_i=\gamma_i\hat x_i+\beta_iη_ℓ = η_top γ^(L − ℓ)
Lower layers (general features) change more slowly when fine-tuning.
Worked example: top = 0.0001, g = 0.9, L = 12, l = 6 → 5.31441e-05 ✓ checked
See also: exponential learning-rate decay
\eta_\ell=\eta_{\text{top}}\,\gamma^{L-\ell}squared and hinge variants of the discriminator loss
Worked example: dr = 0.8, df = 0.2 → [0.04, 1.4] ✓ checked
See also: GAN objective
\text{LSGAN: }\tfrac12\mathbb E(D(x)-1)^2+\tfrac12\mathbb E\,D(G(z))^2;\qquad \text{hinge: }\mathbb E\max(0,1-D(x))+\mathbb E\max(0,1+D(G(z)))std = √(1/fan_in)
Keeps activations at unit variance for SELU / tanh networks.
Worked example: fi = 100 → 0.1 ✓ checked
See also: He (Kaiming) initialisation · Xavier (Glorot) initialisation
\operatorname{Var}(w)=\frac1{n_{in}},\qquad \text{orthogonal: }W^{\top}W=Iy = W x + b
where W ∈ ℝ^(out×in)
Worked example: W = [[1, 2], [3, 4]], x = [1, 1], b = [0, 1] → [3, 8] ✓ checked
See also: parameter counts
\mathbf y=W\mathbf x+\mathbf bφ(q)ᵀ(Σ φ(k)vᵀ) / φ(q)ᵀ(Σ φ(k))
where φ = positive feature map, e.g. elu + 1
Associativity gives O(T) time and a recurrent form.
Worked example: phiq = [1, 2], phik = [[1, 0], [0, 1]], v = [[1], [3]] → 2.33333 ✓ checked
See also: scaled dot-product attention
\operatorname{Attn}(Q,K,V)_i=\frac{\phi(q_i)^{\top}\sum_j\phi(k_j)v_j^{\top}}{\phi(q_i)^{\top}\sum_j\phi(k_j)}Lip(f) ≤ Π ‖W_ℓ‖₂ · Π Lip(σ_ℓ)
ReLU has Lip 1. Loose, but it is the basis of spectral-norm and Lipschitz-constrained training.
Worked example: s = [2, 1.5, 0.5] → 1.5 ✓ checked
See also: spectral norm · Lipschitz continuity
\operatorname{Lip}(f)\le\prod_{\ell}\|W_\ell\|_2\cdot\prod_\ell\operatorname{Lip}(\sigma_\ell)W′ = W + (α/r)·BA; params = r(d + k)
A 4096×4096 layer at r = 8 trains 65 536 parameters instead of 16.8 million.
Worked example: d = 4096, k = 4096, r = 8 → [65536, 1.67772e+07] ✓ checked
See also: Eckart–Young theorem (low-rank approximation)
W'=W+\frac\alpha r\,BA,\quad B\in\mathbb R^{d\times r},\ A\in\mathbb R^{r\times k},\quad \text{trainable params}=r(d+k)cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ; hₜ = oₜ ⊙ tanh(cₜ)
where f, i, o = forget, input, output gates
The additive update lets gradients flow over long spans.
Worked example: f = 0.5, i = 0.5, cp = 2, ct = 1 → 1.5 ✓ checked
See also: GRU hidden update · sigmoid (logistic function)
c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t,\qquad h_t=o_t\odot\tanh(c_t)forget, input and output gates plus a candidate cell
Then cₜ = f ⊙ cₜ₋₁ + i ⊙ c̃ and hₜ = o ⊙ tanh(cₜ).
Worked example: z = 0 → [0.5, 0] ✓ checked
See also: LSTM cell state update
f=\sigma(W_f[h,x]+b_f),\ i=\sigma(W_i[h,x]+b_i),\ o=\sigma(W_o[h,x]+b_o),\ \tilde c=\tanh(W_c[h,x]+b_c)hidden-layer learning rate ∝ 1/width
Hyper-parameters tuned on a small model transfer to a large one.
Worked example: lr0 = 0.01, w0 = 256, w1 = 4096 → 0.000625 ✓ checked
See also: He (Kaiming) initialisation
\eta_{\text{hidden}}\propto\frac1{\text{width}},\qquad \text{output multiplier}\propto\frac1{\text{width}}aggregate messages from neighbours, then update
Sum, mean and max are the usual permutation-invariant aggregators.
Worked example: nb = [1, 2, 3] → [6, 2, 3] ✓ checked
See also: graph convolution (GCN)
m_v=\bigoplus_{u\in N(v)}\psi(h_u,h_v,e_{uv}),\qquad h_v'=\phi(h_v,m_v)Mish = x tanh(softplus x); h-swish = x·ReLU6(x + 3)/6
Worked example: x = 1 → [0.865098, 0.666667] ✓ checked
See also: Swish / SiLU · GELU
\operatorname{Mish}(x)=x\tanh\big(\ln(1+e^x)\big),\qquad \text{hard-swish}(x)=x\frac{\operatorname{ReLU6}(x+3)}6top-k gate over experts; balance loss α N Σ fᵢ Pᵢ
where f = fraction of tokens routed to expert i, P = mean router probability
Minimised when routing is uniform.
Worked example: f = [0.25, 0.25, 0.25, 0.25], P = [0.25, 0.25, 0.25, 0.25], al = 0.01, N = 4 → 0.01 ✓ checked
See also: softmax
g=\operatorname{softmax}(W_rx),\quad y=\sum_{i\in\text{top-}k}g_i\,E_i(x),\qquad L_{\text{bal}}=\alpha N\sum_{i=1}^{N}f_iP_ix̃ = λxᵢ + (1 − λ)xⱼ; ỹ likewise
Worked example: lam = 0.7, xi = 1, xj = 3 → 1.6 ✓ checked
See also: label smoothing
\tilde x=\lambda x_i+(1-\lambda)x_j,\quad \tilde y=\lambda y_i+(1-\lambda)y_j,\quad \lambda\sim\mathrm{Beta}(\alpha,\alpha)MFU = (6·N·tokens/sec) / peak FLOPs
40–60% is excellent for large-scale training.
Worked example: N = 7e+09, tps = 3000, peak = 3.12e+14 → 0.403846 ✓ checked
See also: training compute estimate
\mathrm{MFU}=\frac{6ND/\Delta t}{P_{\text{peak}}\cdot n_{\text{GPU}}}bytes = N × bits/8
Add the KV cache and activations for inference memory.
Worked example: N = 7e+09, bits = 16 → 14 ✓ checked
See also: training memory per parameter (mixed-precision Adam) · affine quantisation
\text{bytes}=N\cdot\frac{\text{bits}}8,\qquad \text{e.g. }7\text{B parameters at 16 bits}=14\ \text{GB}d_k = d_model / h; MHA = Concat(head₁…head_h) W^O
where h = number of heads
Worked example: d = 512, h = 8 → 64 ✓ checked
See also: scaled dot-product attention
d_k=d_v=\frac{d_{\text{model}}}{h},\quad \operatorname{MHA}=\operatorname{Concat}(\text{head}_1,\dots,\text{head}_h)W^{O}Θ(x, x′) = ∇_θ f(x)·∇_θ f(x′)
In the infinite-width limit it stays constant, and training is kernel regression with Θ.
Worked example: g1 = [1, 2], g2 = [0.5, -1] → -1.5 ✓ checked
See also: Gram matrix · kernel ridge regression
\Theta(x,x')=\nabla_\theta f(x;\theta)^{\top}\nabla_\theta f(x';\theta)adding N(0, σ²) input noise ≡ an L2 penalty σ²‖w‖²
Bishop (1995): Tikhonov regularisation is equivalent to training with noise.
Worked example: w = [1, 2], s2 = 0.1 → 0.5 ✓ checked
See also: L2 penalty (weight decay) · ridge regression (closed form)
\mathbb E\big[(\mathbf w^{\top}(\mathbf x+\boldsymbol\varepsilon)-y)^2\big]=(\mathbf w^{\top}\mathbf x-y)^2+\sigma^2\|\mathbf w\|^2log p(x) = log p(z) + log|det ∂z/∂x|
where z = f(x)
Layers are chosen so the Jacobian determinant is cheap (triangular, coupling).
Worked example: x = 1, a = 2 → -2.22579 ✓ checked
See also: change of variables (Jacobian)
\log p_X(x)=\log p_Z\big(f(x)\big)+\log\Big|\det\frac{\partial f}{\partial x}\Big|merge two blocks' running max, normaliser and output
Lets attention be computed block by block without materialising the T×T matrix.
Worked example: x1 = [1, 2], x2 = [3, 0.5] → 3.46077 ✓ checked
See also: scaled dot-product attention · stable softmax and log-sum-exp
m'=\max(m,\tilde m),\quad \ell'=e^{m-m'}\ell+e^{\tilde m-m'}\tilde\ell,\quad O'=\frac{e^{m-m'}\ell\,O+e^{\tilde m-m'}\tilde\ell\,\tilde O}{\ell'}D*(x) = p_data(x) / (p_data(x) + p_g(x))
At the optimum p_g = p_data, D* = ½.
Worked example: pd = 0.6, pg = 0.2 → 0.75 ✓ checked
See also: GAN objective
D^{*}(x)=\frac{p_{\text{data}}(x)}{p_{\text{data}}(x)+p_g(x)}E_Q R ≤ E_Q R̂ + √((KL(Q‖P) + ln(2√n/δ))/(2n))
Non-vacuous bounds for neural networks have been obtained this way.
Worked example: kl = 50, n = 10000, d = 0.05 → 0.053988 ✓ checked
See also: Kullback–Leibler divergence · generalisation bound for a finite class
\mathbb E_{h\sim Q}R(h)\le\mathbb E_{h\sim Q}\hat R(h)+\sqrt{\frac{D_{\mathrm{KL}}(Q\|P)+\ln\frac{2\sqrt n}\delta}{2n}}dense: (in+1)·out conv: (kₕk_w·c_in + 1)·c_out
Bias counted once per output.
Worked example: i = 784, o = 10, kh = 3, kw = 3, ci = 64, co = 128 → [7850, 73856] ✓ checked
See also: linear (fully connected) layer · convolution output size
\text{dense: }(n_{in}+1)\,n_{out}\qquad \text{conv: }(k_hk_wc_{in}+1)\,c_{out}Σ (n_{ℓ−1} + 1) n_ℓA 784-128-10 network has 101 770 parameters.
Worked example: n = [784, 128, 10] → 101770 ✓ checked
See also: parameter counts
\sum_{\ell}(n_{\ell-1}+1)\,n_\ell,\qquad \text{forward FLOPs}\approx2\times\text{parameters}L(N,D) = E + A/N^α + B/D^β
where N parameters, D training tokens; E, A, B, α, β fitted constants
The Chinchilla form; constants depend on the data and architecture.
Worked example: E = 1.7, A = 400, B = 400, a = 0.3, b = 0.3, N = 1e+09, D = 1e+10 → 2.8981 ✓ checked
See also: training compute estimate
L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}n_out = ⌊(n − k)/s⌋ + 1
Worked example: n = 32, k = 2, s = 2 → 16 ✓ checked
See also: convolution output size
n_{out}=\Big\lfloor\frac{n-k}{s}\Big\rfloor+1,\qquad \text{global average pool: }\frac1{HW}\sum_{h,w}x_{hw}loss falls as a power law in parameters, data and compute
Doubling N multiplies the reducible loss by 2^(−α).
Worked example: a = 0.076 → 0.948684 ✓ checked
See also: parametric scaling law
L(N)=\Big(\frac{N_c}{N}\Big)^{\alpha_N},\quad L(D)=\Big(\frac{D_c}{D}\Big)^{\alpha_D},\quad L(C)=\Big(\frac{C_c}{C}\Big)^{\alpha_C}f = max(0, x) + a·min(0, x); ∂f/∂a = min(0, x)
where a learned
Worked example: x = -2, a = 0.25 → [-0.5, -2] ✓ checked
See also: ReLU and leaky ReLU
f(x)=\max(0,x)+a\min(0,x),\qquad \frac{\partial f}{\partial a}=\min(0,x)R ≤ R̂ + 2ℜₙ(H) + √(ln(1/δ)/(2n))
where ℜₙ = Rademacher complexity
Data-dependent; for linear classes ℜ ≤ BR/√n.
Worked example: rad = 0.02, dl = 0.05, n = 10000 → 0.0522387 ✓ checked
See also: VC generalisation bound
R(h)\le\hat R(h)+2\mathfrak R_n(\mathcal H)+\sqrt{\frac{\ln(1/\delta)}{2n}}r_l = r_{l−1} + (k_l − 1)·Π_{i<l} sᵢWorked example: k = [3, 3, 3], s = [1, 2, 2] → 9 ✓ checked
See also: dilated convolution · convolution output size
r_l=r_{l-1}+(k_l-1)\prod_{i<l}s_iz = μ + σ ⊙ ε, ε ~ N(0, I)
Moves the randomness out of the graph so gradients flow through μ and σ.
Worked example: mu = 1, s = 2, e = 0.5 → 2 ✓ checked
See also: evidence lower bound (VAE)
z=\mu+\sigma\odot\varepsilon,\quad \varepsilon\sim\mathcal N(0,I)divide positive logits (multiply negative ones) by θ > 1
Worked example: z = 2, th = 1.2 → 1.66667 ✓ checked
See also: top-k, top-p and temperature sampling
z_i\leftarrow\begin{cases}z_i/\theta&z_i>0\\ z_i\theta&z_i<0\end{cases}\quad(\text{for tokens already generated})scale residual outputs by 1/√(2L)
where L = number of residual blocks
Keeps the variance of the residual stream from growing with depth (GPT-2).
Worked example: L = 48 → 0.102062 ✓ checked
See also: He (Kaiming) initialisation
\text{scale the last layer of each residual branch by }\frac1{\sqrt{2L}}R = r(x, y) − β log(π/π_ref)
Keeps the policy close to the reference model.
Worked example: r = 2, b = 0.1, lp = -5, lpref = -6 → 1.9 ✓ checked
See also: direct preference optimisation (DPO) · PPO clipped objective
R(x,y)=r_\phi(x,y)-\beta\,\ln\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}yᵢ = xᵢ / √(mean(x²) + ε) · gᵢ
where g = learned gain
LayerNorm without the mean subtraction; used in modern LLMs.
Worked example: x = [3, 4], eps = 1e-12 → [0.848528, 1.13137] ✓ checked
See also: layer normalisation
y_i=\frac{x_i}{\sqrt{\tfrac1d\sum_j x_j^2+\epsilon}}\,g_irotate each coordinate pair by m·θᵢ
where m = position
Relative position enters the attention score through the rotation difference.
Worked example: m = 1, th = 1.5708, x = [1, 0] → [6.12323e-17, 1] ✓ checked
See also: sinusoidal positional encoding · 2-D rotation matrix
\begin{pmatrix}x_{2i}'\\ x_{2i+1}'\end{pmatrix}=\begin{pmatrix}\cos m\theta_i&-\sin m\theta_i\\ \sin m\theta_i&\cos m\theta_i\end{pmatrix}\begin{pmatrix}x_{2i}\\ x_{2i+1}\end{pmatrix},\quad \theta_i=10000^{-2i/d}Attn(Q,K,V) = softmax( QKᵀ/√d_k ) V
where d_k = key dimension
The 1/√d_k keeps dot products from saturating the softmax.
Worked example: Q = [[1, 0]], K = [[1, 0], [0, 1]], V = [[1, 2], [3, 4]] → [[1.66048, 2.66048]] ✓ checked
See also: multi-head attention dimensions · sinusoidal positional encoding · softmax
\operatorname{Attn}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)VFLOPs ≈ 4T²d per layer; attention matrix memory O(T²)
where T = sequence length, d = model width
FlashAttention keeps the maths but tiles it so memory is O(T).
Worked example: T = 2048, d = 4096 → 6.87195e+10 ✓ checked
See also: online softmax (FlashAttention rescaling)
\text{FLOPs}\approx4T^2d\ \ (QK^{\top}\text{ and }AV),\qquad \text{memory}=O(T^2)PE(pos,2i) = sin(pos/10000^(2i/d)); PE(pos,2i+1) = cos(…)
Worked example: pos = 1, i = 0, d = 4 → [0.841471, 0.540302] ✓ checked
See also: scaled dot-product attention
PE_{(pos,2i)}=\sin\!\frac{pos}{10000^{2i/d}},\quad PE_{(pos,2i+1)}=\cos\!\frac{pos}{10000^{2i/d}}J = diag(p) − ppᵀ
Symmetric, rank n − 1, rows sum to zero.
Worked example: z = [0, 0] → [[0.25, -0.25], [-0.25, 0.25]] ✓ checked
See also: gradient of softmax cross-entropy
\frac{\partial p_i}{\partial z_j}=p_i(\delta_{ij}-p_j),\qquad J=\operatorname{diag}(p)-pp^{\top}accept with probability min(1, p/q)
where q = draft model, p = target model
Output distribution is exactly the target's; speed-up comes from verifying several drafted tokens in one pass.
Worked example: p = 0.2, q = 0.5 → 0.4 ✓ checked
See also: importance sampling
P(\text{accept token }x)=\min\Big(1,\frac{p(x)}{q(x)}\Big),\qquad \text{on rejection resample from }\operatorname{norm}\big((p-q)_+\big)p_ℓ = 1 − (ℓ/L)(1 − p_L)
where layer ℓ survives with probability p_ℓ
Worked example: l = 10, L = 20, pL = 0.5 → 0.75 ✓ checked
See also: inverted dropout
p_\ell=1-\frac\ell L(1-p_L)FFN(x) = (SiLU(xW₁) ⊙ xW₃)W₂
Gated FFN used in LLaMA-style models; d_ff ≈ (8/3)d keeps parameters equal to a 4d ReLU FFN.
Worked example: d = 4096 → [1.3421e+08, 1.34218e+08] ✓ checked
See also: gated linear unit · Swish / SiLU
\operatorname{FFN}(x)=\big(\operatorname{SiLU}(xW_1)\odot xW_3\big)W_2,\qquad \text{params}=3\,d\,d_{ff}entropy of softmax(z/T) grows with T
Worked example: z = [2, 0] → 1 ✓ checked
See also: softmax with temperature · Shannon entropy
T\to0:\ \text{one-hot};\qquad T\to\infty:\ \text{uniform};\qquad H(T)\ \text{increasing in }Ttemperature, top-k and nucleus (top-p) truncation before renormalising
T → 0 is greedy decoding.
Worked example: p = [0.5, 0.3, 0.1, 0.1], p0 = 0.75 → 2 ✓ checked
See also: softmax with temperature · beam search with length normalisation
p_i\propto e^{z_i/T};\quad \text{top-}k\text{: keep the }k\text{ largest};\quad \text{top-}p\text{: smallest set with }\sum p\ge p_0C ≈ 6 N D
where N = parameters, D = training tokens; FLOPs
About 2ND forward + 4ND backward.
Worked example: N = 1e+09, D = 2e+10 → 1.2e+20 ✓ checked
See also: parametric scaling law
C\approx 6\,N\,D≈ 16 bytes per parameter, before activations
ZeRO shards these across devices.
Worked example: N = 7e+09 → 112 ✓ checked
See also: transformer parameter count
\underbrace{2}_{\text{bf16 weights}}+\underbrace{2}_{\text{grads}}+\underbrace{4}_{\text{fp32 master}}+\underbrace{4+4}_{\text{Adam }m,v}=16\ \text{bytes/parameter}N ≈ 12·L·d² + V·d
where L layers, width d, vocabulary V
A quick estimate; GPT-3 (L = 96, d = 12288) gives about 175B.
Worked example: L = 96, d = 12288 → 1.73946e+11 ✓ checked
See also: parametric scaling law · training compute estimate
N\approx12\,L\,d^2+V\,d,\qquad \text{(attention }4d^2+\text{FFN }8d^2\text{ per layer)}n_out = (n − 1)s − 2p + k + output_padding
The inverse of the convolution size formula; used in decoders and GANs.
Worked example: n = 16, s = 2, p = 1, k = 4, po = 0 → 32 ✓ checked
See also: convolution output size
n_{out}=(n-1)s-2p+k+p_{out}KL = −½ Σⱼ (1 + log σⱼ² − μⱼ² − σⱼ²)
where q = N(μ, σ²), prior N(0, 1)
Worked example: mu = [0.5], logvar = [0] → 0.125 ✓ checked
See also: KL divergence between two Gaussians · evidence lower bound (VAE)
D_{\mathrm{KL}}=-\tfrac12\sum_j\big(1+\log\sigma_j^2-\mu_j^2-\sigma_j^2\big)hₜ = tanh(W_h hₜ₋₁ + W_x xₜ + b)
Worked example: Wh = 0.5, Wx = 1, h = 1, x = 0.5, b = 0 → 0.761594 ✓ checked
See also: LSTM cell state update · GRU hidden update
h_t=\tanh(W_hh_{t-1}+W_xx_t+b)‖∂h_T/∂h_t‖ ≤ (‖W‖γ)^(T − t)
where γ = bound on the activation derivative
Below 1 the gradient vanishes, above 1 it explodes: the motivation for LSTMs, gradient clipping and orthogonal initialisation.
Worked example: w = 0.9, g = 1, n = 50 → 0.00515378 ✓ checked
See also: LSTM cell state update · gradient norm clipping
\Big\|\frac{\partial h_T}{\partial h_t}\Big\|\le\big(\|W\|\,\gamma\big)^{T-t}R ≤ R̂ + √((d(ln(2n/d) + 1) + ln(4/δ))/n)
where d = VC dimension
Worked example: d = 10, n = 10000, dl = 0.05 → 0.0950742 ✓ checked
See also: generalisation bound for a finite class · Rademacher generalisation bound
R(h)\le\hat R(h)+\sqrt{\frac{d\big(\ln\frac{2n}d+1\big)+\ln\frac4\delta}{n}}reconstruction + codebook loss + commitment loss
where sg = stop-gradient
Worked example: rec = 0.5, ze = 1, e = 0.6, b = 0.25 → 0.7 ✓ checked
See also: evidence lower bound (VAE)
\mathcal L=\|x-\hat x\|^2+\|\operatorname{sg}[z_e]-e\|^2+\beta\|z_e-\operatorname{sg}[e]\|^2critic loss + λ(‖∇D(x̂)‖ − 1)²
The critic estimates the Wasserstein-1 distance; the penalty enforces 1-Lipschitzness.
Worked example: dfake = 0.5, dreal = 1.5, gn = 1.2, lam = 10 → -0.6 ✓ checked
See also: Wasserstein-1 distance (1-D) · GAN objective
L_D=\mathbb E[D(\tilde x)]-\mathbb E[D(x)]+\lambda\,\mathbb E\big[(\|\nabla_{\hat x}D(\hat x)\|_2-1)^2\big]w = g v/‖v‖; W̄ = W/σ_max(W)
Spectral norm makes a layer 1-Lipschitz, which stabilises GAN discriminators.
Worked example: W = [[3, 0], [0, 1]] → 1 ✓ checked
See also: spectral norm · Lipschitz continuity
\mathbf w=g\frac{\mathbf v}{\|\mathbf v\|},\qquad \bar W=\frac{W}{\sigma(W)}std = √(2 / (fan_in + fan_out))
For tanh/sigmoid networks.
Worked example: fi = 100, fo = 100 → 0.1 ✓ checked
See also: He (Kaiming) initialisation
w\sim\mathcal N\!\left(0,\tfrac{2}{n_{in}+n_{out}}\right)L = E[log p(x|z)] − β·KL
where β > 1 encourages disentanglement
Worked example: rec = -10, kl = 2, b = 4 → -18 ✓ checked
See also: evidence lower bound (VAE)
\mathcal L=\mathbb E_q[\log p(x\mid z)]-\beta\,D_{\mathrm{KL}}\big(q(z\mid x)\,\|\,p(z)\big)the TD error trains the critic and weights the actor's gradient
Worked example: r = 1, g = 0.9, vn = 2, v = 1.5 → 1.3 ✓ checked
See also: temporal-difference error · REINFORCE with a baseline
\delta=r+\gamma V(s')-V(s),\qquad \theta\leftarrow\theta+\alpha_\pi\,\delta\,\nabla\ln\pi_\theta(a\mid s),\qquad w\leftarrow w+\alpha_V\,\delta\,\nabla V_w(s)A = Q − V; E_π[A] = 0
Worked example: q = 3, v = 2.5 → 0.5 ✓ checked
See also: generalised advantage estimation
A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),\qquad \mathbb E_{a\sim\pi}A^\pi(s,a)=0V^π(s) = Σₐ π(a|s) Σₛ′ P(s′|s,a) [ r + γV^π(s′) ]
Worked example: p = [0.5, 0.5], r = [1, 0], v = [2, 4], g = 0.9 → 3.2 ✓ checked
See also: Bellman optimality equation · temporal-difference error
V^{\pi}(s)=\sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a)\big[r+\gamma V^{\pi}(s')\big]Q*(s,a) = r + γ Σ P(s′|s,a) max_a′ Q*(s′,a′)
Worked example: r = 1, g = 0.9, qn = 5 → 5.5 ✓ checked
See also: Q-learning update
Q^{*}(s,a)=r+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q^{*}(s',a')y = r + γQ′(s′, μ′(s′)); target ← τ·online + (1 − τ)·target
where τ ≈ 0.005
Worked example: r = 1, g = 0.99, qn = 2, tau = 0.005, phi = 1, phip = 0 → [2.98, 0.005] ✓ checked
See also: exponential moving average · TD3 clipped double-Q target
y=r+\gamma Q_{\phi'}\big(s',\mu_{\theta'}(s')\big),\qquad \phi'\leftarrow\tau\phi+(1-\tau)\phi'Gₜ = Σₖ γᵏ rₜ₊ₖ
where γ ∈ [0,1) = discount factor
Worked example: r = [1, 1, 1], g = 0.5 → 1.75 ✓ checked
See also: Bellman expectation equation
G_t=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}b = (r + γz − V_min)/Δz; split mass between the neighbouring atoms
Worked example: r = 1, g = 0.9, z = 2, vmin = -10, dz = 1 → [12, 13, 0.8] ✓ checked
See also: DQN loss
b=\frac{\mathcal T z_j-V_{\min}}{\Delta z},\quad \mathcal Tz_j=r+\gamma z_j\ (\text{clipped}),\quad \text{split mass between atoms }\lfloor b\rfloor,\lceil b\rceilchoose the action with one table, evaluate it with the other
Removes the maximisation bias of Q-learning.
Worked example: qa = 2, qb_at_argmax = 3, a = 0.5, r = 1, g = 0.9 → 2.85 ✓ checked
See also: Q-learning update
Q_A(s,a)\leftarrow Q_A+\alpha\big[r+\gamma Q_B\big(s',\arg\max_{a'}Q_A(s',a')\big)-Q_A\big]L = (r + γ max Q_target(s′, a′) − Q(s, a))²
where θ⁻ = periodically updated target network
Worked example: r = 1, g = 0.99, qt = 5, q = 4 → 3.8025 ✓ checked
See also: Q-learning update · temporal-difference error
L=\mathbb E\Big[\Big(r+\gamma\max_{a'}Q_{\theta^-}(s',a')-Q_\theta(s,a)\Big)^2\Big]H_eff = 1/(1 − γ)
γ = 0.99 looks about 100 steps ahead.
Worked example: g = 0.99 → 100 ✓ checked
See also: discounted return
H_{\text{eff}}=\frac1{1-\gamma}Âₜ = Σₗ (γλ)ˡ δₜ₊ₗ
where λ trades bias against variance
Worked example: d = [1, 1], gl = 0.5 → 1.5 ✓ checked
See also: temporal-difference error · PPO clipped objective
\hat A_t=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\,\delta_{t+l}preferences follow the policy-gradient rule
Worked example: H = [0, 0], a = 0, R = 1, Rb = 0, al = 0.1 → [0.05, -0.05] ✓ checked
See also: policy gradient (softmax policy)
H_a\leftarrow H_a+\alpha(R-\bar R)(1-\pi_a),\quad H_b\leftarrow H_b-\alpha(R-\bar R)\pi_b\ (b\ne a)no consistent algorithm beats this logarithmic regret
Worked example: mu = 0.5, ms = 0.7 → 2.29419 ✓ checked
See also: UCB1
\liminf_{T\to\infty}\frac{\mathbb E[\mathrm{Regret}_T]}{\ln T}\ge\sum_{i:\mu_i<\mu^*}\frac{\mu^*-\mu_i}{D_{\mathrm{KL}}(\mu_i\|\mu^*)}trajectory probability ∝ exp(reward); gradient = expert features − expected features
Worked example: th = [1, 0], f = [[1, 0], [0, 1]] → [0.731059, 0.268941] ✓ checked
See also: optimal entropy-regularised policy
P(\tau\mid\theta)=\frac{\exp\big(\boldsymbol\theta^{\top}\mathbf f(\tau)\big)}{Z(\theta)},\qquad \nabla_\theta\mathcal L=\tilde{\mathbf f}-\mathbb E_\theta[\mathbf f]p* = (d − c)/(a − b − c + d); value = (ad − bc)/(a − b − c + d)
Matching pennies: p* = ½, value 0.
Worked example: a = 1, b = -1, c = -1, d = 1 → [0.5, 0] ✓ checked
See also: Shapley value
p^*=\frac{d-c}{a-b-c+d},\qquad v=\frac{ad-bc}{a-b-c+d}\quad\text{for payoffs }\begin{pmatrix}a&b\\c&d\end{pmatrix}Gₜ⁽ⁿ⁾ = Σ γᵏ rₜ₊ₖ₊₁ + γⁿ V(sₜ₊ₙ)
n = 1 is TD(0), n = ∞ is Monte Carlo.
Worked example: r = [1, 1, 1], g = 0.9, v = 5 → 6.355 ✓ checked
See also: λ-return and eligibility traces · temporal-difference error
G_t^{(n)}=\sum_{k=0}^{n-1}\gamma^kr_{t+k+1}+\gamma^nV(s_{t+n})ρ = π/b; Retrace truncates the ratio at 1
Worked example: pi = 0.6, b = 0.3, lam = 1 → [2, 1] ✓ checked
See also: importance sampling
\rho_t=\frac{\pi(a_t|s_t)}{b(a_t|s_t)},\qquad \text{Retrace: }c_s=\lambda\min\big(1,\tfrac{\pi}{\mu}\big)π* ∝ π_ref exp(Q*/β)
The softmax policy; also the closed-form solution of KL-regularised RL (RLHF).
Worked example: q = [1, 2], b = 1 → [0.268941, 0.731059] ✓ checked
See also: soft actor–critic · RLHF reward with KL penalty
\pi^*(a\mid s)\propto\pi_{\text{ref}}(a\mid s)\,\exp\big(Q^*(s,a)/\beta\big)V^π = (I − γP_π)⁻¹ r_π
Exact for small MDPs; iterative methods scale to large ones.
Worked example: P = [[0.9, 0.1], [0.2, 0.8]], r = [1, 0], g = 0.9 → [7.56757, 4.86486] ✓ checked
See also: Bellman expectation equation · value iteration
V^{\pi}=(I-\gamma P_\pi)^{-1}\mathbf r_\pi∇J = E[ ∇ log π(a|s) · Â ]; ∇_z log π(a) = eₐ − π
REINFORCE. For a softmax policy the gradient on the logits is the one-hot action minus the policy.
Worked example: pi = [0.5, 0.5], a = 0 → [0.5, -0.5] ✓ checked
See also: PPO clipped objective · generalised advantage estimation
\nabla_\theta J=\mathbb{E}\big[\nabla_\theta\log\pi_\theta(a\mid s)\,\hat A\big];\quad \nabla_z\log\pi(a)=\mathbf e_a-\piact greedily with respect to Q^π
Alternating evaluation and improvement converges to the optimal policy.
Worked example: q = [1, 3, 2] → 1 ✓ checked
See also: value iteration
\pi'(s)=\arg\max_aQ^{\pi}(s,a)\ \Rightarrow\ V^{\pi'}\ge V^{\pi}F = γΦ(s′) − Φ(s)
Ng, Harada and Russell (1999): adding F leaves the optimal policy unchanged.
Worked example: g = 0.9, ps = 1, p2 = 3 → 1.7 ✓ checked
See also: Bellman optimality equation
F(s,s')=\gamma\Phi(s')-\Phi(s)L = E[ min( r·Â, clip(r, 1−ε, 1+ε)·Â ) ]
where r = probability ratio, ε ≈ 0.2
Stops the policy moving too far in one update.
Worked example: r = 1.5, A = 2, e = 0.2 → 2.4 ✓ checked
See also: policy gradient (softmax policy) · generalised advantage estimation
L^{\text{CLIP}}=\mathbb{E}\big[\min\big(r_t\hat A_t,\ \operatorname{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t\big)\big],\ r_t=\frac{\pi_\theta(a|s)}{\pi_{\text{old}}(a|s)}Q(s,a) ← Q + α [ r + γ max Q(s′,·) − Q ]
where α = learning rate
Off-policy temporal-difference control.
Worked example: q = 2, a = 0.5, r = 1, g = 0.9, qn = 5 → 3.75 ✓ checked
See also: temporal-difference error · Bellman optimality equation
Q(s,a)\leftarrow Q(s,a)+\alpha\big[r+\gamma\max_{a'}Q(s',a')-Q(s,a)\big]∇J = E[∇ln π · (G − b)]
Any baseline that does not depend on the action leaves the gradient unbiased and can cut the variance.
Worked example: G = 10, b = 8, dlogpi = 0.5 → 1 ✓ checked
See also: policy gradient (softmax policy) · actor–critic
\nabla_\theta J=\mathbb E\big[\nabla_\theta\ln\pi_\theta(a\mid s)\,(G-b(s))\big]Q ← Q + α(r + γQ(s′, a′) − Q)
where a′ = the action actually taken next
On-policy; Q-learning replaces Q(s′, a′) by max.
Worked example: q = 2, a = 0.5, r = 1, g = 0.9, qn = 4 → 3.3 ✓ checked
See also: Q-learning update
Q(s,a)\leftarrow Q(s,a)+\alpha\big[r+\gamma Q(s',a')-Q(s,a)\big]maximum-entropy objective; soft value = α log Σ exp(Q/α)
As α → 0 the soft value tends to max Q.
Worked example: q = [1, 2], al = 0.5 → 2.06346 ✓ checked
See also: log-sum-exp · optimal entropy-regularised policy
J=\mathbb E\sum_t\big[r_t+\alpha\,\mathcal H(\pi(\cdot|s_t))\big],\qquad V(s)=\alpha\ln\sum_a\exp\big(Q(s,a)/\alpha\big)M = (I − γT)⁻¹; V = M r
where T = state-transition matrix under the policy
Separates the environment's dynamics from the reward.
Worked example: T = [[0.5, 0.5], [0, 1]], g = 0.9 → [[1.81818, 8.18182], [0, 10]] ✓ checked
See also: policy evaluation as a linear system
M=(I-\gamma T)^{-1},\qquad V=M\mathbf ry = r + γ min(Q₁′, Q₂′)(s′, μ′(s′) + clipped noise)
The minimum of two critics counters overestimation.
Worked example: r = 1, g = 0.99, q1 = 2, q2 = 1.6 → 2.584 ✓ checked
See also: DDPG target and Polyak averaging · double Q-learning
y=r+\gamma\min\big(Q_1',Q_2'\big)\big(s',\mu'(s')+\operatorname{clip}(\varepsilon,-c,c)\big)δₜ = rₜ + γV(sₜ₊₁) − V(sₜ)
Worked example: r = 1, g = 0.9, vn = 2, v = 1.5 → 1.3 ✓ checked
See also: Q-learning update · generalised advantage estimation
\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)sample each arm's success rate from its posterior; play the best sample
Matches the Lai–Robbins lower bound in practice.
Worked example: al = 1, be = 1, s = 7, f = 3 → 0.666667 ✓ checked
See also: UCB1 · Beta–binomial conjugate update
\theta_i\sim\mathrm{Beta}(\alpha_i+s_i,\ \beta_i+f_i),\quad a_t=\arg\max_i\theta_imaximise the importance-weighted advantage within a KL ball
PPO replaces the constraint with clipping.
Worked example: ratio = 1.2, A = 2 → 2.4 ✓ checked
See also: PPO clipped objective · Kullback–Leibler divergence
\max_\theta\ \mathbb E\Big[\frac{\pi_\theta(a|s)}{\pi_{\text{old}}(a|s)}\hat A\Big]\quad\text{s.t.}\quad\mathbb E\big[D_{\mathrm{KL}}(\pi_{\text{old}}\|\pi_\theta)\big]\le\deltachoose argmax x̄ᵢ + c√(ln t/nᵢ)
where c = √2 in the classic analysis
Optimism in the face of uncertainty: regret O(√(KT ln T)).
Worked example: xbar = 0.5, t = 100, n = 10, c = 1.41421 → 1.45971 ✓ checked
See also: Thompson sampling (Bernoulli) · regret of online gradient descent
a_t=\arg\max_i\Big(\bar x_i+c\sqrt{\frac{\ln t}{n_i}}\Big)select argmax Q̄ + exploration bonus
where N = parent visits, n = child visits, P = prior
PUCT (AlphaZero) uses a policy-network prior.
Worked example: Q = 0.5, N = 100, n = 10, c = 1.4, P = 0.3 → [1.45006, 0.881818] ✓ checked
See also: UCB1
\mathrm{UCT}=\bar Q+c\sqrt{\frac{\ln N}{n}},\qquad \mathrm{PUCT}=\bar Q+c\,P(a)\frac{\sqrt N}{1+n}V ← max_a Σ P (r + γV)
A γ-contraction: error shrinks by γ per sweep.
Worked example: r = [1, 2], v = 3, g = 0.9 → 4.7 ✓ checked
See also: Bellman optimality equation · policy improvement
V_{k+1}(s)=\max_a\sum_{s'}P(s'\mid s,a)\big[r(s,a,s')+\gamma V_k(s')\big]G^λ = (1 − λ) Σ λⁿ⁻¹ G⁽ⁿ⁾; trace e ← γλe + ∇V
Interpolates between TD(0) (λ = 0) and Monte Carlo (λ = 1).
Worked example: lam = 0.5, g = [1, 2, 3] → 1.75 ✓ checked
See also: n-step return · generalised advantage estimation
G_t^\lambda=(1-\lambda)\sum_{n=1}^{\infty}\lambda^{n-1}G_t^{(n)},\qquad e\leftarrow\gamma\lambda e+\nabla V,\ \ \theta\leftarrow\theta+\alpha\,\delta\,e(a + b)/2 ≥ √(ab)
Equality iff a = b.
Worked example: a = 1, b = 4 → 0.5 ✓ checked
See also: Jensen's inequality
\frac{a+b}{2}\ \ge\ \sqrt{ab}\quad(a,b\ge0)(x + y)ⁿ = Σₖ C(n,k) xⁿ⁻ᵏ yᵏ
Worked example: n = 3, x = 1, y = 2 → 27 ✓ checked
See also: binomial distribution · combinations and permutations
(x+y)^n=\sum_{k=0}^{n}\binom nk x^{n-k}y^{k}log_b x = ln x / ln b
Bits vs nats: log₂ x = ln x / ln 2.
Worked example: x = 8, b = 2 → 3 ✓ checked
See also: Shannon entropy
\log_b x=\frac{\ln x}{\ln b}C(n,k) = n! / (k!(n−k)!); P(n,k) = n! / (n−k)!
Worked example: n = 5, k = 2 → [10, 20] ✓ checked
See also: binomial theorem · binomial distribution
\binom nk=\frac{n!}{k!\,(n-k)!},\qquad P(n,k)=\frac{n!}{(n-k)!}(f ∗ g)[n] = Σₘ f[m] g[n−m]
Convolution in time is multiplication in frequency.
Worked example: f = [1, 2, 3], g = [0, 1, 0.5] → [0, 1, 2.5, 4, 1.5] ✓ checked
See also: discrete Fourier transform · convolution output size
(f*g)[n]=\sum_m f[m]\,g[n-m]Xₖ = Σₙ xₙ e^(−2πi kn/N)
The FFT computes it in O(N log N).
Worked example: x = [1, 0, 0, 0] → [1, 1, 1, 1] ✓ checked
See also: Euler's formula and identity · discrete convolution
X_k=\sum_{n=0}^{N-1}x_n\,e^{-2\pi i kn/N}e^(iθ) = cos θ + i sin θ; e^(iπ) + 1 = 0
Worked example → 1.22465e-16 ✓ checked
See also: discrete Fourier transform
e^{i\theta}=\cos\theta+i\sin\theta,\qquad e^{i\pi}+1=0∫ e^(−x²) dx = √π
Gives the normalising constant of the normal density.
Worked example → 1.77245 ✓ checked
See also: normal (Gaussian) density
\int_{-\infty}^{\infty}e^{-x^2}\,dx=\sqrt\piΣ a rᵏ = a(1 − rⁿ)/(1 − r); to ∞: a/(1 − r)
Worked example: a = 1, r = 0.5, n = 3 → 1.75 ✓ checked
See also: discounted return
\sum_{k=0}^{n-1}ar^{k}=a\,\frac{1-r^{n}}{1-r},\qquad \sum_{k=0}^{\infty}ar^k=\frac{a}{1-r}\ (|r|<1)‖f(x) − f(y)‖ ≤ L ‖x − y‖
where L = Lipschitz constant
Bounds how fast the output can change; the basis of stability and robustness analysis.
Worked example: x = 1, y = 3 → 2 ✓ checked
See also: spectral norm
\|f(x)-f(y)\|\le L\,\|x-y\|x = (−b ± √(b² − 4ac)) / 2a
where ax² + bx + c = 0
Real roots iff b² − 4ac ≥ 0.
Worked example: a = 1, b = -3, c = 2 → [1, 2] ✓ checked
x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}n! ≈ √(2πn) (n/e)ⁿ
Relative error about 1/(12n).
Worked example: n = 10 → 3.5987e+06 ✓ checked
See also: combinations and permutations
n!\approx\sqrt{2\pi n}\left(\frac ne\right)^{n}No formula matches. Try another word, or clear the field filter.
Tell us what you need and we will add it, with LaTeX, symbols and a worked example. It goes straight to the Excyst team.