Normalized Advantage Function (NAF)

Continuous-control Q-learning via a quadratic advantage function — an alternative to actor-critic approaches like DDPG for handling continuous action spaces without a separate policy network.

Reference

• Title: Normalized Advantage Function (NAF): A Deep Dive into Continuous Control in Reinforcement Learning
• Author: Shivang Shrivastav (blog explainer, not the original paper)
• Published: Dec 2024 (Medium)
• Link: https://shivang-ahd.medium.com/normalized-advantage-functions-a-deep-dive-into-continuous-control-in-reinforcement-learning-23240eee21f4
• Original method: Gu et al., “Continuous Deep Q-Learning with Model-based Acceleration” (ICML 2016) — NAF was introduced there; worth tracking down the original paper separately.

In one line

NAF sidesteps the “max over continuous actions” problem in Q-learning by forcing the Q-function into a quadratic form, Q(s,a) = V(s) + A(s,a), with A(s,a) = -1/2 (a-μ(s))^T P(s) (a-μ(s)), so the action that maximizes Q is always just the learned mean action μ(s) — no separate actor network needed, unlike DDPG.

My notes

TBD — fill in as I read.

Key equations / method

• Advantage: A(s,a) = Q(s,a) − V(s)
• NAF’s quadratic form: A(s,a) = -1/2 (a-μ(s))^T P(s) (a-μ(s))
• μ(s): mean/optimal action for state s (parametric, learned)
• P(s): positive-definite matrix shaping the advantage falloff away from the mean — larger entries along a dimension mean bigger advantage changes for deviations there (drives exploration)
• Optimal action is simply μ(s) by construction, so no inner-loop optimization or actor network is needed to compute max_a Q(s,a)

Relevance to thesis

Useful point of comparison for the continuous-action-space extension of QVIRL/ValueWalk: NAF shows one way to make Q-learning tractable in continuous spaces via a parametric/quadratic Q-function rather than an actor-critic split. Worth comparing against how DDPG and IRL methods like AVRIL handle the same continuous-action problem.

Questions for Ondřej

• Does the quadratic/unimodal assumption on A(s,a) limit NAF’s expressiveness in ways that matter for a Bayesian IRL extension (e.g. multimodal reward posteriors)?