David Silver’s Lecture 7. Optimize the policy directly, without going through value. The likelihood-ratio trick and the score function, REINFORCE, baselines and advantage, actor-critic, natural policy gradient, and a summary that ties six faces into one.
David Silver’s Lecture 6. When states are too many to write down in a table, approximate the value with a handful of parameters. The incremental methods that learn features by gradient descent, mountain car and the bootstrapping debate, Baird’s counterexample where off-policy diverges, and the batch methods and DQN that pool experience.
David Silver’s Lecture 5. Finding the optimal policy without a model. Why action values rather than state values, the two doors and ε-greedy and GLIE, Sarsa and Sarsa(λ) on the windy gridworld, importance sampling and off-policy learning, Q-learning through cliff walking, and the correspondence between DP and TD.
Just as Newton’s F=ma describes the physical world, information geometry describes how AI learns. An intuitive guide for beginners.
David Silver’s Lecture 4. Estimate the value of a policy from experience alone, with no model of the environment. Monte-Carlo in Blackjack, temporal-difference learning on the drive home, the unified view of bootstrapping and sampling, and TD(λ) with eligibility traces.
David Silver’s Lecture 3. When the MDP is fully known, solve the Bellman equations by iteration to compute the optimal policy. Policy evaluation, policy iteration, and value iteration through a small grid world example.
David Silver’s Lecture 2. The Markov decision process is the container that holds the reinforcement learning problem, built up one layer at a time, from the Markov chain through the reward process to the decision process, alongside Silver’s ‘student’ example.
David Silver’s lecture 1. What sets reinforcement learning apart from other kinds of learning, reward and state, and the three pieces that make up an agent (policy, value function, model).
The two best free ways into reinforcement learning: Sutton & Barto’s textbook (free PDF) and David Silver’s course (free videos). Where to get them, to open this series.