Ferkans — Interactive Telecom Tutor

When You Are Allowed to Look Into the Future

If we record the observation sequence $\{Y_m\}$ off-line and then process it, we can use samples from both the past and the future of time $n$ to estimate $X_n$ . This is the setting of smoothing: image denoising, audio restoration, seismic trace processing. The non-causal Wiener filter is the best linear smoother, and its transfer function is so simple that it deserves to be memorized.

Theorem: Non-Causal Wiener Filter

Let $\{X_n\}, \{Y_n\}$ be jointly WSS with spectra ${P_x}_{x}(f), {P_x}_{y}(f), P_{xy}(f)$ and ${P_x}_{y}(f) > 0$ for all $f$ . The MMSE non-causal linear estimator $\hat{X}_n = \sum_{k \in \mathbb{Z}} h[k] Y_{n-k}$ has transfer function $\boxed{\;\check{h}_{\text{nc}}(f) = \frac{P_{xy}(f)}{P_y(f)}\;}$ and the non-causal MMSE per sample is $\sigma_{\text{nc}}^2 = \int_{-1/2}^{1/2} \left[P_x(f) - \frac{|P_{xy}(f)|^2}{P_y(f)}\right] df.$

Read the formula as a frequency-by-frequency matched filter. At each frequency $f$ , we have a "cross-correlation" $P_{xy}(f)$ between the desired signal and the observation, and an "observation power" $P_y(f)$ . The Wiener gain is the cross-spectrum normalized by the observation spectrum. Where the signal is strong relative to the observation — meaning $|P_{xy}(f)|^2 \approx P_x(f) P_y(f)$ — the gain is close to one. Where the signal is drowned in noise — meaning $|P_{xy}(f)|^2 \ll P_x(f) P_y(f)$ — the gain is close to zero. The filter attenuates the bands where listening would hurt you.

Proof

Write the Wiener-Hopf equation for $\mathcal{K} = \mathbb{Z}$

From TOrthogonality Principle (Wiener-Hopf Equations) with $\mathcal{K} = \mathbb{Z}$ the orthogonality condition becomes $\sum_k h[k] r_{yy}[\ell - k] = r_{xy}[\ell]$ for every $\ell \in \mathbb{Z}$ . The left-hand side is the convolution $(h * r_{yy})[\ell]$ .

Take the DTFT

The Fourier transform turns convolution into multiplication: $\check{h}_{\text{nc}}(f) P_y(f) = P_{xy}(f)$ . Dividing by $P_y(f) > 0$ gives the boxed formula.

Compute the MMSE in the frequency domain

By TMMSE of the Wiener Estimator the MMSE is $\sigma^2 = r_{xx}[0] - \sum_k h[k] r_{xy}^*[k]$ . Using Parseval: $\sum_k h[k] r_{xy}^*[k] = \int_{-1/2}^{1/2} \check{h}_{\text{nc}}(f) P_{xy}^*(f)\,df = \int_{-1/2}^{1/2} \frac{|P_{xy}(f)|^2}{P_y(f)}\,df$ . Similarly $r_{xx}[0] = \int_{-1/2}^{1/2} P_x(f)\,df$ . Subtracting gives the stated formula.

,

Definition:
Signal in Additive Independent Noise

The canonical application is $Y_n = X_n + Z_n$ where $\{Z_n\}$ is a zero-mean WSS noise process independent of $\{X_n\}$ . In this case $r_{xy}[k] = r_{xx}[k]$ , so $P_{xy}(f) = P_x(f), \qquad P_y(f) = P_x(f) + P_z(f),$ and the non-causal Wiener filter takes the classical form $\check{h}_{\text{nc}}(f) = \frac{P_x(f)}{P_x(f) + P_z(f)}.$

Theorem: MMSE for Signal in Additive Independent Noise

Under the setting of DSignal in Additive Independent Noise, $\sigma_{\text{nc}}^2 = \int_{-1/2}^{1/2} \frac{P_x(f)\, P_z(f)}{P_x(f) + P_z(f)}\,df.$

The integrand is the harmonic mean of $P_x(f)$ and $P_z(f)$ at each frequency (up to a factor of 2). It is always less than $\min(P_x(f), P_z(f))$ : the Wiener filter never does worse than simply keeping the signal or throwing it away at each frequency, because the optimum choice is to do something in between.

Proof

Substitute into the MMSE formula

From TNon-Causal Wiener Filter with $P_{xy}(f) = P_x(f)$ and $P_y(f) = P_x(f) + P_z(f)$ : $\sigma_{\text{nc}}^2 = \int \left[P_x(f) - \frac{P_x(f)^2}{P_x(f) + P_z(f)}\right] df = \int \frac{P_x(f)(P_x(f) + P_z(f)) - P_x(f)^2}{P_x(f) + P_z(f)}\,df = \int \frac{P_x(f) P_z(f)}{P_x(f) + P_z(f)}\,df.$

Wiener Filter Frequency Response

AR(1) signal $X_n = a X_{n-1} + U_n$ observed in additive white noise $Z_n$ . Visualize the non-causal and causal Wiener filters on the same axes. High SNR: both gains approach 1 at signal peaks. Low SNR: both gains collapse toward zero. Compare the phase of the causal filter (implicit here — only magnitude is shown).

Parameters

AR coefficient a0.8

SNR (dB)10

MMSE vs SNR: Causal, Non-Causal, and the Signal Variance Floor

For the AR(1)+noise problem, trace the three MMSE curves as SNR varies. At high SNR both filters drive the MMSE to zero; at low SNR both are limited by the signal variance. The gap between causal and non-causal MMSE is the price of real-time processing.

Parameters

AR coefficient a0.8

Min SNR (dB)-10

Max SNR (dB)30

Wiener Denoising in the Time Domain

Simulate an AR(1) realization, add white noise, and compare the noisy observation, the true signal, the non-causal estimate (via DFT), and the causal estimate (via the recursive first-order filter derived from spectral factorization). Vary the AR coefficient and SNR to see the filter adapt.

Parameters

AR coefficient a0.85

SNR (dB)5

Number of samples200

Random seed7

Example: AR(1) Signal in White Noise: Closed-Form Non-Causal Wiener Filter

Let $X_n = a X_{n-1} + U_n$ where $U_n$ is white with variance $\sigma_u^2$ and $|a| < 1$ , and let $Y_n = X_n + Z_n$ with $Z_n$ independent white noise of variance $\sigma_z^2$ . Derive $\check{h}_{\text{nc}}(f)$ and the non-causal MMSE in closed form.

Solution

Compute the signal PSD

The AR(1) has transfer function $1/(1 - a e^{-j2\pi f})$ driven by white noise, so $P_x(f) = \sigma_u^2 / |1 - a e^{-j2\pi f}|^2 = \sigma_u^2 / (1 - 2a\cos(2\pi f) + a^2)$ .

Form the observation PSD

$P_y(f) = P_x(f) + \sigma_z^2$ . The cross-PSD is $P_{xy}(f) = P_x(f)$ since $X$ and $Z$ are independent.

Write the Wiener gain

$\check{h}_{\text{nc}}(f) = \dfrac{P_x(f)}{P_x(f) + \sigma_z^2} = \dfrac{\sigma_u^2}{\sigma_u^2 + \sigma_z^2(1 - 2a\cos(2\pi f) + a^2)}.$ This is a real-valued, frequency-selective attenuator. At $f = 0$ and $f = 1/2$ the signal PSD takes its extreme values, and the gain is correspondingly large or small.

Compute the MMSE

Using TMMSE for Signal in Additive Independent Noise: $\sigma_{\text{nc}}^2 = \int_{-1/2}^{1/2} \dfrac{P_x(f) \sigma_z^2}{P_x(f) + \sigma_z^2}\,df$ . In the high-SNR limit $\sigma_z^2 \to 0$ we get $\sigma_{\text{nc}}^2 \approx \sigma_z^2 \cdot \int 1/P_x(f) \,df \cdot \sigma_z^2 / \text{...} \to 0$ (linearly in $\sigma_z^2$ ). In the low-SNR limit $\sigma_u^2 \ll \sigma_z^2$ the MMSE approaches the signal variance $\sigma_u^2/(1-a^2)$ — the filter effectively gives up.

Common Mistake: $P_y(f) = 0$ Breaks the Non-Causal Formula

Mistake:

Applying $\check{h}_{\text{nc}}(f) = P_{xy}(f)/P_y(f)$ at frequencies where $P_y(f) = 0$ .

Correction:

If the observation PSD vanishes at some frequency — for example, $Y$ is band-limited and $f$ is outside the band — then there is no observation power at that frequency and the formula is indeterminate. The correct interpretation: set $\check{h}_{\text{nc}}(f) = 0$ at such frequencies (we have no information, so estimate zero). The integrand in the MMSE formula should then be replaced by $P_x(f)$ at those frequencies (we lose the signal power entirely).

Key Takeaway

The non-causal Wiener filter has a one-line formula: $\check{h}_{\text{nc}}(f) = P_{xy}(f)/P_y(f)$ . It is a frequency-selective attenuator that keeps bands of high SNR and suppresses bands of low SNR. The MMSE is the integral of $P_x(f) - |P_{xy}(f)|^2/P_y(f)$ , which is never negative by Cauchy-Schwarz on the cross-spectrum.

⚠️Engineering Note

Implementing the Non-Causal Wiener Filter with the DFT

In practice the non-causal filter is implemented block-by-block on a finite observation segment using the DFT. Take an $N$ -point DFT of the observation, multiply by the sampled Wiener gain $\check{h}_{\text{nc}}(k/N)$ , and inverse-DFT. This is asymptotically the optimal smoother, but it introduces circular-convolution artifacts at the block boundaries. Overlap-save and overlap-add reduce the boundary effects. For very long recordings the gain can be precomputed once and reused — the per-sample cost of the filter is then $O(\log N)$ by FFT, compared with the $O(N^2)$ cost of solving the Toeplitz Wiener-Hopf system directly.

Practical Constraints

•
Block-based DFT implementation requires $P_y(f) > 0$ on the DFT grid.
•
Finite-block effects appear at boundaries; overlap-save recommended for streaming.

The Non-Causal Wiener Filter

When You Are Allowed to Look Into the Future

Theorem: Non-Causal Wiener Filter

Write the Wiener-Hopf equation for $\mathcal{K} = \mathbb{Z}$

Take the DTFT

Compute the MMSE in the frequency domain

Definition: Signal in Additive Independent Noise

Theorem: MMSE for Signal in Additive Independent Noise

Substitute into the MMSE formula

Wiener Filter Frequency Response

Parameters

MMSE vs SNR: Causal, Non-Causal, and the Signal Variance Floor

Parameters

Wiener Denoising in the Time Domain

Parameters

Example: AR(1) Signal in White Noise: Closed-Form Non-Causal Wiener Filter

Compute the signal PSD

Form the observation PSD

Write the Wiener gain

Compute the MMSE

Common Mistake: Py(f)=0P_y(f) = 0Py​(f)=0 Breaks the Non-Causal Formula

Key Takeaway

Implementing the Non-Causal Wiener Filter with the DFT

Definition:
Signal in Additive Independent Noise

Common Mistake: $P_y(f) = 0$ Breaks the Non-Causal Formula