References & Further Reading

References

  1. A. D. Wyner, The common information of two dependent random variables, 1975

    Introduces common information as the minimum rate to render two variables conditionally independent. A fundamental measure of shared structure beyond mutual information.

  2. R. Ahlswede and J. Korner, Source coding with side information and a converse for degraded broadcast channels, 1975

    Establishes the rate region for source coding with a helper and connects it to broadcast channel capacity.

  3. S. S. Pradhan and K. Ramchandran, Distributed source coding using syndromes (DISCUS): Design and construction, 2003

    The practical breakthrough for Slepian-Wolf coding using LDPC syndromes.

  4. R. Puri, A. Majumdar, and K. Ramchandran, PRISM: A new robust video coding architecture based on distributed compression principles, 2002

    Practical distributed video coding architecture based on Wyner-Ziv and Slepian-Wolf principles.

  5. A. Aaron and B. Girod, Compression with side information using turbo codes, 2002

    Turbo-code-based distributed video coding, complementing the LDPC-based DISCUS approach.

  6. N. Tishby, F. C. Pereira, and W. Bialek, The information bottleneck method, 1999

    Introduces the information bottleneck as a rate-distortion framework for extracting relevant information. A foundational paper connecting information theory to representation learning.

  7. G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, Information bottleneck for Gaussian variables, 2005

    Analytical solution of the information bottleneck for jointly Gaussian variables.

  8. D. P. Kingma and M. Welling, Auto-encoding variational Bayes, 2014

    Introduces the variational autoencoder (VAE), whose loss function is a rate-distortion objective with log-loss distortion.

  9. A. A. Alemi, B. Poole, I. Fischer, J. V. Dillon, R. A. Saurous, and K. Murphy, Fixing a broken ELBO, 2018

    Formalizes the rate-distortion interpretation of VAEs and the connection to the information bottleneck.

  10. R. Shwartz-Ziv and N. Tishby, Opening the black box of deep neural networks via information, 2017

    Proposes that deep networks undergo information compression during training. Influential but partially challenged by subsequent work.

Further Reading

For deeper exploration of the connections between source coding, distributed coding, and machine learning.

  • Gacs-Korner common information

    P. Gacs and J. Korner, 'Common information is far less than mutual information,' Problems of Control and Information Theory, 1973.

    The other notion of common information, measuring extractable shared randomness rather than generatable shared randomness.

  • Neural image compression

    J. Balle, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, 'Variational image compression with a scale hyperprior,' ICLR 2018.

    State-of-the-art learned image compression using VAE-like architectures that directly optimize rate-distortion.

  • Deep learning and information theory

    A. Zaidi, I. Estella-Aguerri, and S. Shamai, 'On the information plane of autoencoders,' IEEE ISIT 2020.

    Rigorous analysis of the information-theoretic properties of autoencoders and their connection to rate-distortion theory.

ExercisesCh 9 →