Expectation and Variance for Continuous RVs

Why Expectation and Variance?

A distribution contains complete information about a random variable, but it is often too much information. We need summary statistics: a single number for the "center" of the distribution (expectation) and a single number for the "spread" (variance). These two numbers, together with the distribution family, often suffice for engineering design.

Definition:

Expectation of a Continuous Random Variable

Let XX be a continuous RV with PDF ff. The expectation (or mean) of XX is

E[X]=βˆ«βˆ’βˆžβˆžx f(x) dx,\mathbb{E}[X] = \int_{-\infty}^{\infty} x\,f(x)\,dx,

provided the integral converges absolutely: βˆ«βˆ’βˆžβˆžβˆ£xβˆ£β€‰f(x) dx<∞\int_{-\infty}^{\infty} |x|\,f(x)\,dx < \infty.

The Cauchy distribution f(x)=1Ο€(1+x2)f(x) = \frac{1}{\pi(1 + x^2)} is the canonical example of a distribution whose expectation does not exist: ∫∣x∣/(1+x2) dx=∞\int |x|/(1 + x^2)\,dx = \infty.

Theorem: Law of the Unconscious Statistician (LOTUS)

Let XX be a continuous RV with PDF ff and let g:R→Rg : \mathbb{R} \to \mathbb{R} be a Borel-measurable function. Then

E[g(X)]=βˆ«βˆ’βˆžβˆžg(x) f(x) dx,\mathbb{E}[g(X)] = \int_{-\infty}^{\infty} g(x)\,f(x)\,dx,

provided the integral converges absolutely.

We do not need to find the distribution of Y=g(X)Y = g(X) to compute E[Y]\mathbb{E}[Y]. We can integrate gg directly against the density of XX. This "unconscious" application of the original density is enormously useful in practice.

Proof
Key idea

Consider the induced probability measure PX(A)=∫Af(x) dx\mathbb{P}_X(A) = \int_A f(x)\,dx. Then E[g(X)]=∫Rg(x) dPX(x)=∫g(x)f(x) dx\mathbb{E}[g(X)] = \int_{\mathbb{R}} g(x)\,d\mathbb{P}_X(x) = \int g(x)f(x)\,dx by the definition of integration with respect to a measure. β– \blacksquare

Definition:

Variance of a Continuous Random Variable

The variance of a continuous RV XX with finite second moment is

Var(X)=E[(Xβˆ’E[X])2]=E[X2]βˆ’(E[X])2.\text{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2.

The standard deviation is ΟƒX=Var(X)\sigma_X = \sqrt{\text{Var}(X)}.

The identity Var(X)=E[X2]βˆ’(E[X])2\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2 is the computational workhorse β€” it is almost always easier to compute E[X2]\mathbb{E}[X^2] and E[X]\mathbb{E}[X] separately than to compute E[(Xβˆ’ΞΌ)2]\mathbb{E}[(X - \mu)^2] directly.

Theorem: Tail Integration Formula for Expectation

Let XX be a non-negative continuous RV. Then

E[X]=∫0∞P(X>t) dt=∫0∞[1βˆ’F(t)] dt.\mathbb{E}[X] = \int_0^{\infty} \mathbb{P}(X > t)\,dt = \int_0^{\infty} [1 - F(t)]\,dt.

More generally, if XX takes values in R\mathbb{R}:

E[X]=∫0∞P(X>t) dtβˆ’βˆ«0∞P(X<βˆ’t) dt.\mathbb{E}[X] = \int_0^{\infty} \mathbb{P}(X > t)\,dt - \int_0^{\infty} \mathbb{P}(X < -t)\,dt.

Instead of integrating xβ‹…f(x)x \cdot f(x), we integrate the survival function 1βˆ’F(t)1 - F(t). This is often much easier when the CDF has a simple closed form but the PDF does not (or when we only know tail bounds).

Proof
Exchange order of integration (Tonelli)

For Xβ‰₯0X \geq 0:

∫0∞P(X>t) dt=∫0βˆžβ€‰β£βˆ«0∞1{x>t} f(x) dx dt.\int_0^{\infty} \mathbb{P}(X > t)\,dt = \int_0^{\infty}\!\int_0^{\infty} \mathbf{1}_{\{x > t\}}\,f(x)\,dx\,dt.

By Tonelli's theorem (both integrands are non-negative), we swap the integrals:

=∫0∞f(x)β€‰β£βˆ«0xdt dx=∫0∞x f(x) dx=E[X].β– = \int_0^{\infty} f(x)\!\int_0^{x} dt\,dx = \int_0^{\infty} x\,f(x)\,dx = \mathbb{E}[X]. \quad\blacksquare

Example: Expectation of the Exponential Distribution via Tail Integration

Let X∼Exp(λ)X \sim \text{Exp}(\lambda). Use the tail integration formula to compute E[X]\mathbb{E}[X].

Solution
Survival function

P(X>t)=eβˆ’Ξ»t\mathbb{P}(X > t) = e^{-\lambda t} for tβ‰₯0t \geq 0.

Apply the formula

E[X]=∫0∞eβˆ’Ξ»t dt=[βˆ’1Ξ»eβˆ’Ξ»t]0∞=1Ξ».\mathbb{E}[X] = \int_0^{\infty} e^{-\lambda t}\,dt = \left[-\frac{1}{\lambda}e^{-\lambda t}\right]_0^{\infty} = \frac{1}{\lambda}.

Compare with direct computation

Direct integration gives E[X]=∫0∞xΞ»eβˆ’Ξ»x dx=1/Ξ»\mathbb{E}[X] = \int_0^{\infty} x \lambda e^{-\lambda x}\,dx = 1/\lambda (integration by parts). The tail formula avoids the integration by parts entirely.

Example: Variance of the Uniform Distribution

Compute Var(X)\text{Var}(X) for X∼Uniform[a,b]X \sim \text{Uniform}[a, b].

Solution
Compute $\mathbb{E}[X]$

E[X]=∫abxbβˆ’a dx=a+b2.\mathbb{E}[X] = \int_a^b \frac{x}{b-a}\,dx = \frac{a + b}{2}.

Compute $\mathbb{E}[X^2]$

E[X2]=∫abx2bβˆ’a dx=a2+ab+b23.\mathbb{E}[X^2] = \int_a^b \frac{x^2}{b-a}\,dx = \frac{a^2 + ab + b^2}{3}.

Variance

Var(X)=a2+ab+b23βˆ’(a+b2)2=(bβˆ’a)212.\text{Var}(X) = \frac{a^2 + ab + b^2}{3} - \left(\frac{a+b}{2}\right)^2 = \frac{(b-a)^2}{12}.

Theorem: Linearity and Scaling of Variance

For any random variable XX with finite variance and constants a,b∈Ra, b \in \mathbb{R}:

Var(aX+b)=a2 Var(X).\text{Var}(aX + b) = a^2\,\text{Var}(X).

In particular, adding a constant does not change the variance, and scaling by aa multiplies the variance by a2a^2.

Proof
Direct computation

Let ΞΌ=E[X]\mu = \mathbb{E}[X]. Then E[aX+b]=aΞΌ+b\mathbb{E}[aX + b] = a\mu + b, so

Var(aX+b)=E[(aX+bβˆ’aΞΌβˆ’b)2]=E[a2(Xβˆ’ΞΌ)2]=a2 Var(X).β– \text{Var}(aX + b) = \mathbb{E}[(aX + b - a\mu - b)^2] = \mathbb{E}[a^2(X - \mu)^2] = a^2\,\text{Var}(X). \quad\blacksquare

Common Mistake: Variance Is Not Linear

Mistake:

Writing Var(2X)=2Var(X)\text{Var}(2X) = 2\text{Var}(X).

Correction:

Var(aX)=a2 Var(X)\text{Var}(aX) = a^2\,\text{Var}(X), so Var(2X)=4Var(X)\text{Var}(2X) = 4\text{Var}(X). Variance scales with the square of the constant. Standard deviation scales linearly.

Quick Check

For X∼Exp(3)X \sim \text{Exp}(3), what is ∫0∞P(X>t) dt\int_0^{\infty} \mathbb{P}(X > t)\,dt?

33

1/31/3

1/91/9

infty\\infty

Correction:
1/31/3

By the tail integration formula, ∫0∞P(X>t) dt=E[X]=1/Ξ»=1/3\int_0^{\infty} \mathbb{P}(X>t)\,dt = \mathbb{E}[X] = 1/\lambda = 1/3.

LOTUS (Law of the Unconscious Statistician)

The identity E[g(X)]=∫g(x) f(x) dx\mathbb{E}[g(X)] = \int g(x)\,f(x)\,dx, which allows computing the expectation of a function of XX without first deriving the distribution of g(X)g(X).

Related: Probability Density Function (PDF)

Variance

Var(X)=E[(Xβˆ’E[X])2]\text{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]: the expected squared deviation from the mean. Measures the spread of a distribution.

Key Takeaway

The tail integration formula E[X]=∫0∞P(X>t) dt\mathbb{E}[X] = \int_0^{\infty} \mathbb{P}(X > t)\,dt for non-negative RVs is one of the most useful tricks in probability. It converts an expectation computation into an integral of the survival function, which is often simpler β€” especially when only tail bounds are available.

PDF, CDF, and the Relationship Between ThemCommon Continuous Distributions