The origin of Bayesian IRL — the paper that introduced the reward posterior and the PolicyWalk MCMC sampler. Everything in this line of work (bajgar-2024-valuewalk, main-vi-paper) descends from here. See thesis-hub / reading-roadmap.
Reference
- Title: Bayesian Inverse Reinforcement Learning
- Authors: Deepak Ramachandran, Eyal Amir
- Venue: IJCAI 2007 (Proceedings of the 20th International Joint Conference on Artificial Intelligence), pp. 2586–2591
- Link: https://www.ijcai.org/Proceedings/07/Papers/416.pdf
In one line
Casts IRL as Bayesian inference: treat the reward function as a random variable with a prior, and use the expert’s demonstrations as evidence. The likelihood is a Boltzmann/softmax over the optimal -values — each expert action in state has probability (rationality parameter ). Inference over the resulting reward posterior is done by PolicyWalk, an MCMC (grid random-walk Metropolis) sampler; the posterior mean minimises expected reward loss.
Relationship to other papers
- Foundational ancestor of bajgar-2024-valuewalk. ValueWalk’s key move is to sample in Q-value space rather than reward space as PolicyWalk does here — avoiding repeatedly re-solving the forward MDP. This note is the baseline that framing is defined against.
- Same Bayesian-IRL lineage that main-vi-paper reframes onto variational-inference instead of MCMC.
- Engine: mcmc (PolicyWalk is a Metropolis random walk over a reward grid).
My notes
TBD — fill in as I read.
Key equations / method
- Likelihood:
- Posterior:
- PolicyWalk: MCMC over a discretised reward grid; each step perturbs one reward component, recomputes , accepts/rejects Metropolis-style. (Details TBD.)
Relevance to thesis
TBD — the historical foundation; anchors the “reward-space vs. Q-space sampling” contrast the thesis builds on.