QVIRL Continuous-Action Extension — Directions & Findings

Logged 2026-07-22 from a research-planning session with Claude. Caveat: several derivations below are Claude’s sketches, not published results — verify by hand before relying on them.

The problem

Extending QVIRL to continuous/high-dim actions = fixing three bottlenecks:

1. Partition integral in the likelihood: ∫ e^{βQ(s,a)} da (replaces the discrete softmax sum; breaks the Lu et al. approximation).
2. Continuous argmax in the max-mean / inverse-Bellman step (breaks Lemma 1’s linearization).
3. Policy extraction: argmax of μ_Q (or μ − kσ for risk-aversion) over continuous a.

Viability test for any extension: does Lemma 1’s closed-form reward-posterior propagation (and hence the analytic KL) survive? Paper’s own stopgap (App. B.2: ~100 uniform contrastive actions) dies under curse of dimensionality.

Six candidate directions

1. Quadratic-in-action Q (NAF) — Q = V(s) − ½(a−μ_a)ᵀP(a−μ_a). Argmax and partition both exactly analytic; Boltzmann policy becomes Gaussian N(μ_a, (βP)⁻¹). Weakness: unimodal in actions (QT-Opt quote re grasping). Strengthened by latent-action hybrid (quadratic Q in PLAS-style learned latent → multimodality via decoder; caveat: changes expert-model semantics, decoder Jacobian in likelihood). Bounded actions → erf/truncated-Gaussian terms, still closed form. Refs: Gu et al. 2016; BNAF (NeurIPS 2022); PLAS; Chandak et al. 2019.
2. Actor + importance-weighted contrastives — amortize argmax à la DDPG/TD3; Lemma 1 survives verbatim. Expected baseline; build regardless. Note: naive sampled-set Lu (App. B.2) lacks importance weights — deriving the corrected continuous Lu is a publishable piece. Risk: actor suboptimality and max-mean bias compound (both underestimate V). Refs: DDPG, TD3, IQ-Learn, Van de Wiele 2020.
3. LGCP connection — ∫e^{βQ}da is a log-Gaussian Cox process intensity integral. Jensen: E[−log Z] ≥ −log E[Z], correct direction for ELBO; E[Z] pointwise analytic (log-normal). Composable with 1/2 (likelihood only). Theory-flavored, apparently untouched. Refs: Lloyd/Gunter/Osborne/Roberts ICML 2015; Møller 1998; Probabilistic Numerics.
4. Autoregressive discretization (Q-Transformer / VLA route) — per-dimension bins, autoregressive over action dims; both QVIRL tricks apply unchanged per dimension. Open theory: factorized conditional softmaxes ≠ joint softmax expert model (matters for inverse RL). Refs: Q-Transformer 2023; Metz SDQN; Tang & Agrawal 2020; RT-2/OpenVLA/FAST.
5. Score matching — identity ∇_a log π_E = β∇_a Q* eliminates the partition function (normalizer independent of a). Info-preservation: ∇_a destroys exactly the per-state constant the likelihood never identified. Expected DSM loss under deep-kernel GP posterior = exact closed-form quadratic: ‖βg_θ + (ã−a)/σ²‖² + β²tr V_θ (needs no approximation — cleaner than discrete Lu). Cost: not an ELBO → generalized/Gibbs posterior (Bissiri 2016, Knoblauch GVI); needs MCMC-grounded calibration study. DSM noise σ reinterpretable as “Boltzmann + Gaussian motor noise” expert model. Policy extraction: Langevin on β∇(μ−kσ) → test-time-tunable risk-averse multimodal diffusion policy.
6. Soft-Bellman — V_soft = β⁻¹log∫e^{βQ}; unifies bottlenecks 1+2, but requires reparameterization — the mechanism QVIRL’s authors found destabilizing. Hinges on advisor confirming instability origin. Refs: Haarnoja SQL/SAC; IQ-Learn; Ziebart.

Key finding: EnergyFlow (prior art, verified)

“Recovering Hidden Reward in Diffusion-Based Policies” (EnergyFlow), Ji et al., arXiv:2605.00623, May 2026, code: github.com/sotaagi/EnergyFlow.

• Publishes exactly the inverted-QSM mechanism of Direction 5: DSM score = gradient of expert’s soft Q under max-ent optimality; reward extraction without adversarial training; conservative-field constraint + identifiability/error-propagation bounds; SOTA on RoboMimic and Meta-World.
• Non-Bayesian (point estimate; bounds, not posteriors). Open wedge = the Bayesian layer: GP posterior, closed-form expected DSM loss, calibration, risk-averse policies, active learning.
• Positioning sentence: “EnergyFlow is to our method what IQ-Learn was to QVIRL.”
• Synergy: their conservative-field constraint comes for free from a scalar-Q GP posterior.
• Now mandatory related work for any continuous-action QVIRL route.

Current verdict

• Score matching (D5): lower-risk, lower-novelty-ceiling — mechanism validated at manipulation scale, ready baseline/codebase, but crowded lane (racing risk on “add uncertainty to EnergyFlow”).
• LGCP (D3) and latent-quadratic (D1+7): higher-risk, higher-ceiling — unscooped, derivation risk.
• For the arm goal (active learning + few-demo on robomimic): D4 backbone or latent-quadratic, D2 baseline. Payoffs (uncertainty-driven querying, few-demo) live in the Bayesian layer either way.

Advisor-meeting questions

1. Concurrent work on a Bayesian EnergyFlow?
2. Was reparameterization instability really the source of the closed-form constraint? (gates D5/D6)
3. What happened to the ICLR-version Safety Gymnasium experiments (dropped in RLC 2026 resubmission)?
4. Appetite: fast-solid (D5 delta) vs slow-distinctive (D3/D1+7)?

Artifacts from the session

• qvirl_continuous_actions.md — full study document (QVIRL recap, bottlenecks, six directions, comparison table, phased plan).
• score_matching_for_qvirl.md — Direction 5 deep dive (score matching from first principles, closed-form derivation, generalized-posterior fine print, ordered reading path).
• Both contain unverified Claude derivations (DSM quadratic form, LGCP Jensen bound, info-preservation lemma) — check by hand.

---

Update 2026-07-23 — Directions 7–10 (second session with Claude)

Same caveat as above: derivation sketches are Claude’s, verify by hand. Papers below were verified to exist via web search.

New candidate directions

7. Exact-density inversion (Bayesian CSIL) — model π_E(a|s) with an exact-likelihood model (conditional normalizing flow), then invert soft optimality: βQ = log π_E + βV_soft(s). Partition function is not approximated — it is absorbed into the per-state constant the likelihood never identified (same identifiability fact D5 exploits, but D7 keeps the full log-density instead of differentiating it away). Prior art: CSIL (Watson, Huang, Heess, NeurIPS 2023 spotlight; code github.com/google-deepmind/csil) — BC policy inverted into shaped reward + critic hypothesis space, shaping = Ng-1999 potential Ψ(s). Non-Bayesian → same open wedge as EnergyFlow. Positioning: “CSIL is to D7 what EnergyFlow is to D5.” Selling point vs D5: flow likelihood is exact (change of variables) → potential to keep a true ELBO / coherent posterior instead of Gibbs/generalized posterior. Bonus: ∇ log π_flow is conservative by construction (gradient of an actual scalar) — EnergyFlow’s conservative-field constraint comes for free. Key derivation sketch (verify!): with flow fixed, V_soft(s) = Q(s,a) − β⁻¹log π(a|s) pointwise → V_soft affine in Q → inverse soft Bellman maps GP posterior on Q to Gaussian reward posterior without max-mean linearization. Risks: flow expressiveness/topology, log amplifies density errors in tails (Q → −∞ where flow underestimates), two-stage plug-in vs coherent joint model (see deep-dive doc). Scooping risk concentrated: Watson’s own background is Bayesian last-layer / functional GP priors — check whether Bayesian CSIL is on his roadmap.

8. Stein/SVGD closed-form entropy (contingency upgrade of D6) — S²AC (ICLR 2024, arXiv:2405.00987): policy = parameterized SVGD sampler from EBM over Q; closed-form entropy expression (first-order derivatives + vector products, via invertibility of SVGD update). Gives V_soft = E[Q] + β⁻¹H without Gaussian reparameterization; deterministic transport ≠ the reparam mechanism that destabilized QVIRL — but gated on the same advisor question about instability origin. Open derivation: does anything stay analytic under the GP posterior.

9. Action-space inducing points (App B.2 done properly) — replace fixed uniform contrastive set with SVGP inducing inputs (Titsias 2009; Hensman 2015) whose action locations are variational parameters optimized by the ELBO. Adaptive contrastive set + approximation error accounted for inside the variational bound; discrete Lu runs on the inducing set. DKL+SVGP = standard plumbing, low implementation risk. Arguably replaces D2 as the expected baseline. Aligns with active-learning payoff (inducing locations ≈ where to query).

10. Mixture-of-quadratics Q (multimodal NAF, unverified sketch) — Q = β⁻¹ log Σₖ wₖ(s) exp(β qₖ(s,a)), qₖ quadratic → e^{βQ} is literally a Gaussian mixture: partition exact, Boltzmann policy = GMM (multimodal, no PLAS decoder, no Jacobian in likelihood). Argmax ≈ max over K component modes (cheap) → slots into max-mean. Directly answers the QT-Opt unimodality critique while keeping the viability test. Prior-art check needed before investing (MDN policies / GMM critics are old; composition may be scooped or scooped-by-obviousness).

Component library (bottleneck 2 only, composable)

• Exact continuous argmax: CAQL (ICLR 2020) — max-Q over ReLU network via MIP, globally optimal; ICNN (Amos 2017) — Q concave in a → argmax is a convex program. An ICNN action-head in the deep-kernel mean would make the max-mean step exact; combine with D3/D9/D10 for bottleneck 1.

Related-work ammunition

• Distributional IRL (arXiv, Oct 2025): aleatoric return-distribution counterpart to our epistemic reward posterior — sharpens what the risk-averse story claims. Quotable motivation: states explicitly that BIRL “fails to model continuous action spaces as obtaining the likelihood is computationally intractable.” A 2025 paper asserting the thesis problem is open.
• GPIRL (Levine & Popović 2010-ish): future-work section already promises GP reward variance → conservative policies + querying the expert where variance is high. 15-year-old promissory note that the Bayesian layer cashes — intro-chapter framing device.

Updated verdict

• D7 joins the top tier next to D5: same invert-the-density-model family, but true-ELBO potential converts the biggest logged theoretical liability (generalized posterior / broken lower bound) into a selling point. Trade: flows less battle-tested than diffusion at manipulation scale; EnergyFlow’s validated mechanism is D5’s remaining edge.
• D9 = new expected baseline (replaces D2; barely harder, theoretically cleaner).
• D8 = contingency keyed to advisor’s answer on instability origin. D10 = cheap experiment that could rescue D1.

Advisor questions (additions)

5. Is a Bayesian CSIL on Watson’s (or anyone’s) radar? Advisor’s community overlaps with that circle.

Artifacts (additions)

• density_inversion_for_qvirl.md — D7 deep dive from first principles (soft optimality → inversion identity → CSIL → Bayesian layer, two candidate probabilistic constructions, comparison vs D5/EnergyFlow, failure modes, reading path). Contains unverified Claude derivations — check by hand.

---

Update 2026-08-29 — TODOs from OB meeting

Source: meeting transcript (Czech), summarised by Claude. Transcript cut off mid-answer on VI Gaussian expressivity — last item below.

Mine

• Held-out LL logging — split demos into train/val; log both. Retag all existing ledger LL as training-LL. Blocker for everything else.
• Rerun overnight parameter sweeps with held-out LL.
• K → 1 test — does apprentice return survive with a single (or zero) contrastive action? Prediction: return flat, posterior spread changes. Normalise LL across K before comparing.
• Floor weight: treat as hyperparameter, sweep; note 1/(1+K) scaling and dimension dependence (OB conceded the latter).
• Read inverse-Bellman Q→R mapping in the paper (bijection in discounted MDP; practical version approximates argmax) — needed for reward-posterior LL.
• Multiple val sets: expert-like vs expert+ε-exploration (filter ε-random actions when using for predictive LL). Quantify with seed-to-seed variance across val sets.
• Check Minari expert policy (SAC?) and its temperature → how deterministic are the demos.
• Note HalfCheetah demo return discrepancy (expert ~14k vs random 100-demo sample ~4k) — numbers forgotten, recover.
• Ask study office about PSU (Posilované učení, KP, Bajgar/Pevný) — fall vs spring by programme.

Owed by OB

• Clean main repo, push with full eval wiring (held-out LL, auxiliary data).
• Test hypothesis: more init noise → LL/return correlation improves.
• Timeline for soft Q-learning (SAC-style) switch — changes the likelihood; gates proposal-network work.
• Unfinished answer: reward-posterior LL against Minari rewards under reward shift/gauge (true reward at +100, posterior 0±3 → log-prob → −∞). His idealised-Bayesian argument doesn’t address it; VI Gaussian limitation was where the transcript cut off. Push in writing.

Things OB confirmed / said

• Prior mean irrelevant to behaviour, variance only — except agent-controlled termination (Hopper). Negative reward → incentive to end episode. HalfCheetah fixed-length.
• Held-out LL is gameable when val ≈ train; test-set diversity is the crux. (Consider SBC as cleaner calibration instrument.)
• Auxiliary data (apprentice rollouts into training set) consistently raises val LL, destabilises training; not in simplified code.
• “Posterior spread irrelevant for apprenticeship return” is publishable — implies return-only evaluation of Bayesian IRL is inadequate. Need the positive half (CVaR/risk-averse setting where spread matters).
• Policy head (P) may matter for the max approximation; paper’s improved max uses multiple actions → K problem more central.