This post opens a series working through David MacKay’s Information Theory, Inference, and Learning Algorithms. Before the chapters, a note on why this book, and where to get everything for free.
Why this book
Most courses teach information theory, statistical inference, and machine learning as three separate subjects. MacKay’s move was to treat them as one subject told from three sides. Shannon’s channel coding, Bayesian inference, and neural networks turn out to circle the same question: how to represent and update belief under uncertainty. The book makes that unity concrete, with a Bayesian slant throughout and a well-loved set of hand-drawn figures and exercises.
It is also a joy to read. MacKay writes like a teacher who wants you to see the idea, not admire the notation.
Get the book (free)
MacKay released the full book as a free PDF and it has stayed free. Cambridge University Press published it in September 2003 (640 pages); the author’s own page hosts it for on-screen reading and download.
- Free PDF, about 11 MB: inference.org.uk/itprnn/book.pdf
- Book page, with other formats (djvu, epub, PostScript): inference.org.uk/mackay/itila/book.html
- Table of contents: inference.org.uk/mackay/itila/toc.html
Watch the lectures (free)
In 2012 MacKay’s Cambridge course, “Information Theory, Pattern Recognition, and Neural Networks,” was filmed: sixteen lectures that follow the book closely. They are on YouTube, free.
- YouTube playlist, 16 lectures: Information Theory, Pattern Recognition, and Neural Networks
- Course page, with slides and notes: inference.org.uk/itprnn
Lecture 1, to get a feel for how he teaches:
The plan
From here the series follows the book chapter by chapter, pairing each with the matching lecture. MacKay died in 2016, but he kept these materials open so anyone could learn from them. That is as good a reason as any to start.