OpenAI Spinning Up’s reference writeup on DDPG — the canonical actor-critic approach to continuous control, contrasts with the single-network approach of NAF.
Reference
- Title: Deep Deterministic Policy Gradient
- Source: OpenAI Spinning Up docs (algorithm writeup, not the original paper)
- Link: https://spinningup.openai.com/en/latest/algorithms/ddpg.html
- Original papers:
- Silver et al., “Deterministic Policy Gradient Algorithms” (ICML 2014) — theory
- Lillicrap et al., “Continuous Control with Deep Reinforcement Learning” (ICLR 2016) — the deep RL adaptation (DDPG proper)
In one line
DDPG concurrently learns a Q-function Q_φ(s,a) (off-policy, via the Bellman equation) and a deterministic policy μ_θ(s) that approximates argmax_a Q_φ(s,a), using the policy network as a differentiable stand-in for the otherwise-intractable max over a continuous action space — an actor-critic alternative to NAF’s single quadratic-Q-function approach.
My notes
TBD — fill in as I read.
Key equations / method
- Q-learning side: minimize MSBE loss
L(φ,D) = E[(Q_φ(s,a) − (r + γ(1−d) Q_φtarg(s', μ_θtarg(s'))))²] - Policy side: gradient ascent on
E[Q_φ(s, μ_θ(s))]w.r.t. θ only (Q-params treated as constant) - Uses replay buffer (off-policy) + target networks updated via polyak averaging:
φ_targ ← ρφ_targ + (1−ρ)φ - Exploration: additive Gaussian (or OU) noise on actions at train time, none at test time; random actions for the first
start_steps - Off-policy, only for continuous action spaces; no parallelization in the Spinning Up implementation
Relevance to thesis
Standard baseline for continuous-action RL and a natural comparison point for extending QVIRL/ValueWalk to continuous action spaces — DDPG’s actor-critic split (separate policy network approximating the argmax) is one solution to the same “max over continuous actions” problem that NAF solves differently. Also relevant background for anything downstream that needs a continuous-control policy-learning subroutine (e.g. IRL methods built on top of a Q-learner).
Questions for Ondřej
- For a Bayesian IRL extension, would the actor-critic split of DDPG make posterior inference over rewards easier or harder than NAF’s single-network approach?