Rather than going through value, parameterize the policy $\pi_\theta(a\mid s)$ directly and raise the objective by gradient ascent. It is strong on continuous actions and stochastic policies, and it is the approach that leads into the mainstream of modern reinforcement learning, PPO included.

Lecture slides: PDF

Why work with the policy directly

Value-based methods learn a value function and derive a policy greedily from it. Policy-based methods put the policy directly under parameters and optimize it. Combining the two gives actor-critic.

Working with the policy directly has several advantages. Convergence is smoother, continuous and high-dimensional actions are handled naturally, and above all it can produce a stochastic policy. The drawbacks are that it is prone to local optima, and policy evaluation is inefficient and high-variance.

Silver shows with two examples why a stochastic policy is essential. The first is rock-paper-scissors. Playing deterministically lets the opponent read and exploit you. The only optimum (the Nash equilibrium) is the stochastic policy that plays all three uniformly at 1/3 each. A value-based greedy policy cannot express this optimum at all. The second is the aliased gridworld. When two cells look identical in observation and cannot be told apart, a deterministic policy always picks the same direction in those cells and gets trapped at a dead end on one side. A stochastic policy mixes left and right to escape. This is the point where randomness becomes a remedy under partial observability.

What we maximize

We must first fix an objective. For episodic problems it is the value of the start state, $J_1(\theta) = v_{\pi_\theta}(s_1)$. For never-ending continuing problems, we use the average value weighted by the fraction of time spent in each state when the policy runs long (the stationary distribution $d^{\pi_\theta}(s)$), or the average reward per step. The three objectives take different forms, but remarkably the policy gradient theorem holds in the same shape for all three. So if we derive just one, the rest follow.

Finite differences

The crudest way to obtain the gradient is finite differences. Perturb the parameters one axis at a time, measure how much the objective changes, and approximate the gradient. With $n$ parameters this requires $n$ evaluations, so it is slow and noisy, but the policy need not be differentiable and it is easy to implement. In fact, the fast gait of the four-legged robot AIBO was tuned this way. By perturbing the twelve parameters that define the leg trajectories, the robot found on its own a walk of 291 millimeters per second, faster than the gait humans had tuned by hand. But as the number of parameters grows, finite differences quickly hit a wall, and an analytic gradient becomes necessary.

Monte-Carlo policy gradient

To obtain the gradient analytically, we need one trick. The gradient of the objective produces a sum like $\nabla_\theta \sum_a \pi_\theta(a\mid s)\, Q(s,a)$, which is not an expectation over the policy and cannot be estimated from samples. Using the likelihood-ratio identity $\nabla_\theta \pi_\theta = \pi_\theta \nabla_\theta \log \pi_\theta$ brings it back to an expectation over $\pi_\theta$. The $\nabla_\theta \log \pi_\theta(a\mid s)$ that pops out here is the score function. This is where the log comes from.

$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[ \nabla_\theta \log \pi_\theta(a\mid s)\; q_{\pi_\theta}(s,a) \right]$$

The shape of the score function falls out cleanly for each policy. For a softmax policy over discrete actions, the score is “the feature $\phi(s,a)$ of the chosen action minus the mean feature under the policy.” It means pushing probability in the direction that was better than average. For a Gaussian policy over continuous actions, it is $(a-\mu)\phi/\sigma^2$: if the actual action was above the mean and the outcome was good, it shifts the mean that way.

Putting the actual return $G_t$ in place of $q$ gives REINFORCE, the Monte-Carlo policy gradient. It is unbiased but high-variance, so learning wobbles. As in Silver’s puck-world example, where force is applied continuously, policy gradient works naturally.

Actor-critic

The key to reducing variance is to use a value estimated by a critic in place of $G_t$. The critic solves the problem of evaluating the policy (Lecture 4), and the actor pushes the policy with that evaluation. The two are learned together.

Subtracting a baseline reduces variance further. The key point is that the expectation of the score function is zero. Since $\sum_a \nabla_\theta \pi_\theta(a\mid s) = \nabla_\theta \sum_a \pi_\theta = \nabla_\theta 1 = 0$, subtracting any baseline $B(s)$ that depends only on the state leaves the expected gradient unchanged. Only the variance goes down. The best baseline is the state value $V(s)$, and then $q(s,a) - V(s)$ is exactly the advantage function $A(s,a)$.

There is an even more welcome fact in practice. The expectation of the TD error $\delta = r + \gamma V(s') - V(s)$ is exactly the advantage.

$$\mathbb{E}[\delta \mid s,a] = Q(s,a) - V(s) = A(s,a)$$

So without keeping a separate $Q$ approximator, learning just $V$ gives the advantage unbiasedly. This is the practical standard that leads into A2C.

You might worry that using an approximation in the critic distorts the gradient, but there is a theorem that if the critic is chosen to be “compatible” with the policy (its features match the score function and it minimizes the error), the gradient is exact even with an approximation (compatible function approximation). Also, the spectrum of MC, TD(0), and TD(λ) seen in Lecture 4 applies not only to the critic but to the actor as well. You can choose whether to use the return (high-variance), the TD error (high-bias), or eligibility traces to blend in between as the target for the actor.

Natural policy gradient

One step further. The same policy can be expressed by different parameters, and the vanilla gradient changes direction depending on the parameterization. The natural policy gradient corrects the gradient with the Fisher information matrix so that it moves a fixed distance in policy (distribution) space rather than in parameter space. Remarkably, combined with compatible approximation, the natural gradient direction becomes the critic’s parameter $w$ itself. It means you just push the actor in the critic’s direction. The idea in the later TRPO and PPO of constraining the KL distance so the policy cannot go too far in one step is a direct descendant of this natural gradient.

Six faces, one equation

The lecture ends by tying the various forms of policy gradient into a single table. All of them are the score function multiplied by something.

  • REINFORCE: score × return $G_t$
  • Q Actor-Critic: score × $Q_w(s,a)$
  • Advantage Actor-Critic: score × $A_w(s,a)$
  • TD Actor-Critic: score × $\delta$
  • TD(λ) Actor-Critic: score × $\delta$, together with eligibility traces
  • Natural Actor-Critic: correct the above gradient with the inverse Fisher matrix

What you multiply by sets the bias-variance trade-off. This actor-critic and advantage estimation are the backbone of modern methods like PPO, stabilized with trust regions and clipping. The target flagged in the resources post picks up here.