https://3rdlayer.uk/tags/lecture-notes/Recent content in Lecture Notes on 3rd layerHugo -- 0.157.0en-USSat, 19 Jun 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-10-games/Sat, 19 Jun 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-10-games/David Silver’s Lecture 10. A case study. Best response and Nash equilibrium seen through game theory, minimax search with a binary-linear value function, self-play reinforcement learning, combining minimax with RL, and imperfect-information games like poker. One recipe running through Chinook, Deep Blue, Logistello, TD-Gammon, and Maven.https://3rdlayer.uk/posts/rl-09-exploration-exploitation/Sat, 15 May 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-09-exploration-exploitation/David Silver’s Lecture 9. Use what you know now (exploitation), or find out more (exploration)? This note ties that dilemma into five principles, then follows the multi-armed bandit through regret and lower bounds, UCB, Thompson sampling, and information-state search, and on to contextual bandits and the extension to MDPs.https://3rdlayer.uk/posts/rl-08-integrating-learning-planning/Sat, 17 Apr 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-08-integrating-learning-planning/David Silver’s Lecture 8. Learn a model of the environment from experience, then plan with that model. Model learning as supervised learning, the AB example, Dyna and Dyna-Q+, Monte-Carlo Tree Search and Go, TD search and Dyna-2.https://3rdlayer.uk/posts/rl-07-policy-gradient/Sat, 13 Mar 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-07-policy-gradient/David Silver’s Lecture 7. Optimize the policy directly, without going through value. The likelihood-ratio trick and the score function, REINFORCE, baselines and advantage, actor-critic, natural policy gradient, and a summary that ties six faces into one.https://3rdlayer.uk/posts/rl-06-value-function-approximation/Sat, 13 Feb 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-06-value-function-approximation/David Silver’s Lecture 6. When states are too many to write down in a table, approximate the value with a handful of parameters. The incremental methods that learn features by gradient descent, mountain car and the bootstrapping debate, Baird’s counterexample where off-policy diverges, and the batch methods and DQN that pool experience.https://3rdlayer.uk/posts/rl-05-model-free-control/Sat, 16 Jan 2021 00:00:00 +0000https://3rdlayer.uk/posts/rl-05-model-free-control/David Silver’s Lecture 5. Finding the optimal policy without a model. Why action values rather than state values, the two doors and ε-greedy and GLIE, Sarsa and Sarsa(λ) on the windy gridworld, importance sampling and off-policy learning, Q-learning through cliff walking, and the correspondence between DP and TD.https://3rdlayer.uk/posts/rl-04-model-free-prediction/Sat, 12 Dec 2020 00:00:00 +0000https://3rdlayer.uk/posts/rl-04-model-free-prediction/David Silver’s Lecture 4. Estimate the value of a policy from experience alone, with no model of the environment. Monte-Carlo in Blackjack, temporal-difference learning on the drive home, the unified view of bootstrapping and sampling, and TD(λ) with eligibility traces.https://3rdlayer.uk/posts/rl-03-dynamic-programming/Sat, 14 Nov 2020 00:00:00 +0000https://3rdlayer.uk/posts/rl-03-dynamic-programming/David Silver’s Lecture 3. When the MDP is fully known, solve the Bellman equations by iteration to compute the optimal policy. Policy evaluation, policy iteration, and value iteration through a small grid world example.https://3rdlayer.uk/posts/rl-02-mdp/Sat, 17 Oct 2020 00:00:00 +0000https://3rdlayer.uk/posts/rl-02-mdp/David Silver’s Lecture 2. The Markov decision process is the container that holds the reinforcement learning problem, built up one layer at a time, from the Markov chain through the reward process to the decision process, alongside Silver’s ‘student’ example.https://3rdlayer.uk/posts/rl-01-introduction/Sat, 19 Sep 2020 00:00:00 +0000https://3rdlayer.uk/posts/rl-01-introduction/David Silver’s lecture 1. What sets reinforcement learning apart from other kinds of learning, reward and state, and the three pieces that make up an agent (policy, value function, model).