AVRIL — Scalable Bayesian Inverse Reinforcement Learning (Chan & van der Schaar, 2021)

The variational entry point into scalable BIRL. Closest neighbour to main-vi-paper — read alongside it. Parent: thesis-hub (see reading-roadmap).

Reference

• Title: Scalable Bayesian Inverse Reinforcement Learning
• Method name: AVRIL — Approximate Variational Reward Imitation Learning
• Authors: Alex J. Chan, Mihaela van der Schaar
• Venue: ICLR 2021
• Links: https://arxiv.org/abs/2102.06483 · OpenReview · code (JAX)

In one line

Classic BIRL (ramachandran-2007-bayesian-irl) needs an inner-loop MDP solver, so it doesn’t scale past small tabular problems. AVRIL replaces the intractable posterior with a variational approximation: it jointly trains a reward encoder q_φ(R | s,a) (an amortised posterior over the reward) and a Q-value decoder Q_θ, tied together by a consistency penalty so the reward matches the TD error. Everything is learned fully offline from logged trajectories — no environment interaction — which is what makes it usable in high-stakes settings like clinical decision-making.

My notes

TBD — fill in as I read.

Key equations / method

TBD — the joint (φ, θ) objective: Boltzmann policy term + KL to reward prior + λ·(TD-error consistency term). Write it out once read.

Relevance to thesis

This is the reference point for the VI-based branch of the line we extend. Where bajgar-2024-valuewalk does the Bayesian posterior via HMC in Q-space, AVRIL does it via amortised variational-inference — same “reward-as-latent” framing, different inference engine. Good for understanding what main-vi-paper improves on and where richer priors / continuous-control extensions could plug in.

Questions for Ondřej

• How does AVRIL’s amortised q_φ relate to the variational family used in main-vi-paper?