Non-Bayesian imitation-learning baseline, but shares the Q-space trick that makes bajgar-2024-valuewalk efficient. Parent: thesis-hub.
Reference
- Title: IQ-Learn: Inverse soft-Q Learning for Imitation
- Authors: Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, Stefano Ermon
- Venue: NeurIPS 2021 (Spotlight)
- Links: https://arxiv.org/abs/2106.12142 · OpenReview · code
In one line
Adversarial IL (GAIL / AIRL) frames imitation as a min-max game over separate reward and policy networks — unstable and hard to train. IQ-Learn collapses this into learning a single soft Q-function, which implicitly defines both the reward and the policy, turning the min-max into a single non-adversarial optimisation. Works from very few demos (even one), scales to image envs (SOTA on Atari, used to win a MineRL challenge), and can recover a reward from the environment.
My notes
TBD — fill in as I read.
Key equations / method
TBD — the change of variables from (reward, policy) to a single Q, and the resulting IQ objective / the implied reward recovery r = Q − γ·E[V].
Relevance to thesis
Not Bayesian — no posterior, point estimate only — so it’s a contrast/baseline rather than part of the framework we extend. But the core move (parameterise in Q-space, let Q imply the reward) is exactly the reparameterisation bajgar-2024-valuewalk exploits for cheap Q → reward gradients. Useful for (a) situating the Q-space idea in the wider IL literature, and (b) a strong non-Bayesian point-estimate baseline to compare posterior-based methods against.
Questions for Ondřej
- Is IQ-Learn a baseline worth reproducing, or just context?