EPIC — Equivalent-Policy Invariant Comparison

How do you measure whether a learned reward function is right? Not by training a policy on it — that conflates reward error with optimiser failure. EPIC compares reward functions directly, while ignoring the ways two rewards can differ on paper but mean the same thing in practice. Written as a self-contained chapter: worked numbers throughout, verify everything by hand.

Read this before: any experiment that reports “how good is the recovered reward?” — including the continuous-action work in continuous-action-extension-directions.

---

1. The problem: reward functions are not their numbers

Suppose the ground-truth reward for a navigation task is +1 for reaching the goal. Your IRL algorithm returns +100 for reaching the goal.

Measure the difference numerically — say $L_2$ distance — and you get an enormous error. But the two rewards induce literally the same optimal policy under every transition dynamics. Your algorithm was correct and your metric lied.

This is the Celsius/Fahrenheit problem. 20 °C and 68 °F are different numbers and the same temperature. Comparing reward functions naively is comparing the numerals and ignoring the units.

The alternative that also fails

The obvious fix is to train a policy on the learned reward and see how well it does under the true reward. This is what the field did before EPIC, and it has two defects:

1. It confounds two failures. A bad score could mean the reward is wrong, or it could mean RL failed to optimise a perfectly good reward. You cannot tell which.
2. It only tests one environment. A reward can look fine in the training environment and incentivise catastrophic behaviour in a slightly different deployment environment. Policy-based evaluation cannot see this.

Both defects matter for alignment specifically: a reward model that scores well on-distribution and misgeneralises off-distribution is exactly the failure mode we want an evaluation to catch, not hide.

EPIC’s move is to compare $R_1$ and $R_2$ as functions, with no policy optimisation anywhere in the pipeline — but to do the comparison in a coordinate system where the meaningless differences have been quotiented away.

---

2. The two invariances

Two rewards induce the same optimal policy — under every transition dynamics — if they differ by these two transformations:

2.1 Positive affine transformation

$R^{\prime }(s,a,s^{\prime })=\lambda R(s,a,s^{\prime })+c,\lambda >0$

Scale everything up, shift everything by a constant. Every policy’s return is transformed monotonically, so the ranking over policies is untouched. This is the Celsius/Fahrenheit case.

2.2 Potential shaping (Ng, Harada & Russell, 1999)

This one is subtler, and it is the reason EPIC needs machinery rather than just a correlation.

Pick any function $\Phi :\mathcal{S}\to \mathbb{R}$ assigning a number to each state. Define

$R^{\prime }(s,a,s^{\prime })=R(s,a,s^{\prime })+\gamma \Phi (s^{\prime })-\Phi (s)$

Why does this preserve behaviour? Sum the added terms along a trajectory $s_0\to s_1\to \cdots \to s_T$, with discounting:

${\sum }_{t=0}^{T-1}{\gamma }^t[\gamma \Phi (s_{t+1})-\Phi (s_t)]=\underset{\text{only endpoints survive}}{\underbrace{{\gamma }^T\Phi (s_T)-\Phi (s_0)}}$

Everything telescopes. The interior terms cancel in a chain. So the total added reward depends only on where you started and where you ended — not on the route. Every path from $A$ to $B$ shifts by the same amount, so the ranking of paths is preserved exactly.

The intuition

Shaping is a loan against future reward. $\Phi$ is a “you’re doing well” bonus paid out for entering a state, and immediately clawed back when you leave it. Over any complete journey the loans net out. You can move reward earlier in time — which is genuinely useful for RL training speed — without changing what is being rewarded.

2.3 Worked example — a shaped reward that looks nothing like the original

Three states in a line, goal at C:

graph LR
    A((A)) -->|R| B((B))
    B -->|R| C((C, goal))
    B -->|L| A
    C -->|L| B

True reward $R_1$: $+1$ whenever $s^{\prime }=C$, else $0$.

Shaping potential: $\Phi (A)=0,\Phi (B)=5,\Phi (C)=10$, with $\gamma =1$.

Shaped reward $R_2=R_1+\Phi (s^{\prime })-\Phi (s)$:

transition	$R_1$	$\Phi (s^{\prime })-\Phi (s)$	$R_2$
A →R→ B	0	$5-0$	+5
B →R→ C	1	$10-5$	+6
B →L→ A	0	$0-5$	−5
C →L→ B	0	$5-10$	−5
C →R→ C	1	$10-10$	+1

$R_2$ looks like a completely different objective: it pays out lavishly for moving right and punishes moving left, whereas $R_1$ is indifferent to everything except arriving at C.

Now total up any route from A to C:

• A→B→C: $R_1$ gives $0+1=1$; $R_2$ gives $5+6=11$.
• A→B→A→B→C: $R_1$ gives $0+0+0+1=1$; $R_2$ gives $5-5+5+6=11$.

Always exactly 10 more, on every route, because $\Phi (C)-\Phi (A)=10$. Identical rankings, identical optimal policy.

$\Vert R_1-R_2{\Vert }_2$ is large. The right answer is that the distance is zero.

---

3. The pipeline

graph TD
    R1["R₁ raw"] --> C1["canonicalise C(R₁)"]
    R2["R₂ raw"] --> C2["canonicalise C(R₂)"]
    C1 --> P["Pearson correlation ρ over coverage distribution 𝒟"]
    C2 --> P
    P --> D["d = √((1−ρ)/2) ∈ [0,1]"]
    style C1 fill:#e8f0fe
    style C2 fill:#e8f0fe
    style P fill:#fef3e8
    style D fill:#e8f6ec

step	kills which invariance	mechanism
1. Canonicalise	potential shaping	map each reward to a fixed representative of its shaping class
2. Pearson $\rho$	positive affine	correlation is invariant to $\lambda X+c$, $\lambda >0$
3. $\sqrt{(1-\rho )/2}$	—	turns a similarity into a genuine metric

---

4. Step 1 — Canonicalisation, in full

4.1 The definition

$C(R)(s,a,s^{\prime })=R(s,a,s^{\prime })+\mathbb{E}[\gamma R(s^{\prime },A,S^{\prime })-R(s,A,S^{\prime })-\gamma R(S,A,S^{\prime })]$

where $A\sim {\mathcal{D}}_A$ and $S,S^{\prime }\sim {\mathcal{D}}_S$ are drawn independently of each other and of the transition being evaluated. Lowercase letters are the fixed inputs; capitals are random.

Stare at this and it looks arbitrary. It isn’t. Define

$\boxed{\Psi (s):={\mathbb{E}}_{A\sim {\mathcal{D}}_A,S^{\prime }\sim {\mathcal{D}}_S}[R(s,A,S^{\prime })]}$

Read $\Psi (s)$ as: “sitting in state $s$, if something random happens, what reward do I get on average?” It is a crude one-step goodness score for each state, computed from $R$ alone — no policy, no dynamics, no value iteration. Write $\bar{\Psi }={\mathbb{E}}_{S\sim {\mathcal{D}}_S}[\Psi (S)]$.

Substituting the three expectations:

$C(R)=R+\underset{\text{this is a potential shaping}}{\underbrace{\gamma \Psi (s^{\prime })-\Psi (s)}}-\underset{\text{constant}}{\underbrace{\gamma \bar{\Psi }}}$

The key structural fact

Canonicalisation applies a potential shaping to $R$, using a potential that $R$ itself determines.

That self-reference is the whole trick. Hand it a shaped version of $R$, and the $\Psi$ computed from that shaped version differs in precisely the way needed to undo the shaping. It’s a normalisation that finds its own zero point.

4.2 Proof that shaping cancels

Let $R_2=R_1+\gamma \Phi (s^{\prime })-\Phi (s)$, and let $\bar{\Phi }={\mathbb{E}}_{S\sim {\mathcal{D}}_S}[\Phi (S)]$.

First, how does $\Psi$ transform?

${\Psi }_2(s)=\mathbb{E}[R_1(s,A,S^{\prime })+\gamma \Phi (S^{\prime })-\Phi (s)]={\Psi }_1(s)+\gamma \bar{\Phi }-\Phi (s)$

Note the $-\Phi (s)$ term. That is the piece that will do the cancelling. Averaging again:

${\bar{\Psi }}_2={\bar{\Psi }}_1+\gamma \bar{\Phi }-\bar{\Phi }$

Now expand $C(R_2)$:

$$\begin{array}{rl}C(R_2)& =\underset{R_2}{\underbrace{R_1+\gamma \Phi (s^{\prime })-\Phi (s)}}\\ & +\gamma [{\Psi }_1(s^{\prime })+\gamma \bar{\Phi }-\Phi (s^{\prime })]\\ & -[{\Psi }_1(s)+\gamma \bar{\Phi }-\Phi (s)]\\ & -\gamma [{\bar{\Psi }}_1+\gamma \bar{\Phi }-\bar{\Phi }]\end{array}$$

Collect terms:

term	from	cancels against
$+\gamma \Phi (s^{\prime })$	$R_2$	$-\gamma \Phi (s^{\prime })$ in line 2
$-\Phi (s)$	$R_2$	$+\Phi (s)$ in line 3
$+{\gamma }^2\bar{\Phi }$	line 2	$-{\gamma }^2\bar{\Phi }$ in line 4
$-\gamma \bar{\Phi }$	line 3	$+\gamma \bar{\Phi }$ in line 4

Everything involving $\Phi$ dies. What survives:

$C(R_2)=R_1+\gamma {\Psi }_1(s^{\prime })-{\Psi }_1(s)-\gamma {\bar{\Psi }}_1=C(R_1)■$

Exactly equal — for any $\Phi$, any $\gamma$, any sampling distributions. Two rewards differing by shaping collapse onto the identical canonical form.

4.3 Numerically, on the three-state example

Take ${\mathcal{D}}_S=\text{Uniform}\{A,B,C\}$ and ${\mathcal{D}}_A=\text{Uniform}\{L,R\}$.

For $R_1$ ($+1$ iff $s^{\prime }=C$): since $R_1$ depends only on the destination and $S^{\prime }$ is drawn independently of $s$,

${\Psi }_1(s)=\mathrm{Pr}[S^{\prime }=C]=\frac{1}{3}\text{for every }s,{\bar{\Psi }}_1=\frac{1}{3}$

$C(R_1)=R_1+\gamma \frac{1}{3}-\frac{1}{3}-\gamma \frac{1}{3}=R_1-\frac{1}{3}$

(The $\gamma$ terms cancel because ${\Psi }_1$ is constant here.) So:

$C(R_1)=\{\begin{array}{ll}+2/3& s^{\prime }=C\\ -1/3& \text{otherwise}\end{array}$

For $R_2$ (the shaped one), $\bar{\Phi }=(0+5+10)/3=5$, so ${\Psi }_2(s)=\frac{1}{3}+5-\Phi (s)$:

${\Psi }_2(A)=\frac{16}{3},{\Psi }_2(B)=\frac{1}{3},{\Psi }_2(C)=-\frac{14}{3},{\bar{\Psi }}_2=\frac{1}{3}$

Now check individual entries against $C(R_1)$, using $C(R_2)=R_2+{\Psi }_2(s^{\prime })-{\Psi }_2(s)-{\bar{\Psi }}_2$ (with $\gamma =1$):

transition	$R_2$	$+{\Psi }_2(s^{\prime })$	$-{\Psi }_2(s)$	$-{\bar{\Psi }}_2$	$C(R_2)$	$C(R_1)$
B →R→ C	$6$	$-14/3$	$-1/3$	$-1/3$	$2/3$	$2/3$ ✓
A →R→ B	$5$	$+1/3$	$-16/3$	$-1/3$	$-1/3$	$-1/3$ ✓
C →L→ B	$-5$	$+1/3$	$+14/3$	$-1/3$	$-1/3$	$-1/3$ ✓

Two rewards that looked utterly unrelated — one paying $+6$ where the other pays $+1$ — land on identical canonical forms. This is the payoff.

Do this by hand once

Working the B→R→C row on paper is the single most useful five minutes for internalising EPIC. $6-\frac{14}{3}-\frac{1}{3}-\frac{1}{3}=6-\frac{16}{3}=\frac{18-16}{3}=\frac{2}{3}$.

---

5. Step 2 — Pearson correlation

Canonicalisation killed shaping. It did not kill scaling: the operation is linear in $R$, so

$C(\lambda R+c)=\lambda C(R)\text{for }\lambda >0$

(the constant $c$ washes out entirely, and $\lambda$ passes straight through). Correlation handles the leftover, since $\rho (X,\lambda X+c)=1$ for any $\lambda >0$.

Mechanically: pick a coverage distribution $\mathcal{D}$ over transitions, sample a batch, evaluate both canonical rewards on each sampled transition, correlate the two resulting columns.

Two different distributions, don't conflate them

• ${\mathcal{D}}_S,{\mathcal{D}}_A$ (§4) control how you canonicalise — they define $\Psi$.
• $\mathcal{D}$ (here) controls where you compare — which transitions enter the correlation.

They serve different roles and can be chosen independently. Both are free parameters, and §8 is about why that matters.

5.1 A non-zero distance, computed end to end

Introduce $R_3=+1$ iff $s^{\prime }=B$ — “go to B instead of C”, a genuinely different objective. By the same symmetry argument, $C(R_3)=R_3-\frac{1}{3}$.

Take $\mathcal{D}$ uniform over all $3\times 2\times 3=18$ triples. Both canonical rewards depend only on $s^{\prime }$, and each value of $s^{\prime }$ covers 6 triples, so each carries probability $1/3$:

$s^{\prime }$	$X=C(R_1)$	$Y=C(R_3)$	$XY$
A	$-1/3$	$-1/3$	$+1/9$
B	$-1/3$	$+2/3$	$-2/9$
C	$+2/3$	$-1/3$	$-2/9$

Means: $\mathbb{E}[X]=\frac{1}{3}(-\frac{1}{3}-\frac{1}{3}+\frac{2}{3})=0$, likewise $\mathbb{E}[Y]=0$.

Canonical rewards come out already centred under $\mathcal{D}={\mathcal{D}}_S\times {\mathcal{D}}_A\times {\mathcal{D}}_S$ — that is what the $-\gamma \bar{\Psi }$ term was for. In general (mismatched $\mathcal{D}$) they won’t be, and Pearson subtracts the means anyway.

Covariance: $\text{Cov}(X,Y)=\mathbb{E}[XY]=\frac{1}{3}\left(\frac{1}{9}-\frac{2}{9}-\frac{2}{9}\right)=\frac{1}{3}\cdot \left(-\frac{3}{9}\right)=-\frac{1}{9}$

Variance: $\text{Var}(X)=\frac{1}{3}\left(\frac{1}{9}+\frac{1}{9}+\frac{4}{9}\right)=\frac{1}{3}\cdot \frac{6}{9}=\frac{2}{9}(\text{same for }Y)$

Correlation: $\rho =\frac{-1/9}{\sqrt{(2/9)(2/9)}}=\frac{-1/9}{2/9}=-\frac{1}{2}$

Negative — which is right. “Reach B” and “reach C” actively disagree: each rewards a state the other treats as ordinary.

---

6. Step 3 — Correlation to distance

$\boxed{d_{\text{EPIC}}(R_1,R_2)=\sqrt{\frac{1-\rho }{2}}\in [0,1]}$

For our pair: $\sqrt{(1+0.5)/2}=\sqrt{0.75}\approx 0.87$. Large, as it should be.

6.1 Why this formula and not just $1-\rho$

Because it is genuinely a metric, and there is a clean geometric reason.

Centre both canonical rewards and normalise them to unit length in the $\mathcal{D}$-weighted inner product, giving $\hat{X},\hat{Y}$ on the unit sphere. Then $\rho =⟨\hat{X},\hat{Y}⟩$, and

$\Vert \hat{X}-\hat{Y}{\Vert }^2=\Vert \hat{X}{\Vert }^2+\Vert \hat{Y}{\Vert }^2-2⟨\hat{X},\hat{Y}⟩=2-2\rho$

$⟹\Vert \hat{X}-\hat{Y}\Vert =\sqrt{2-2\rho }=2\sqrt{\frac{1-\rho }{2}}=2d_{\text{EPIC}}$

The geometric picture

EPIC is half the straight-line (chord) distance between two points on a unit sphere.

Every reward function is projected onto that sphere: canonicalisation picks the representative, centring-and-normalising puts it on the surface. Rewards in the same equivalence class map to the same point.

        ρ = 1              ρ = 0             ρ = −1
      X̂ = Ŷ             X̂ ⊥ Ŷ            X̂ = −Ŷ
         ●                 ●                 ●
        ╱ ╲               ╱│╲               ╱ ╲
       ╱   ╲             ╱ │ ╲             ●   ╲
      │  ●  │           │  ●──● │          │ ↕   │
       ╲   ╱             ╲   ╱             ╲   ●
        ╲ ╱               ╲ ╱               ╲ ╱
         ●                 ●                 ●
      d = 0            d ≈ 0.707           d = 1

Scale reference points:

$\rho$	$d_{\text{EPIC}}$	interpretation
$1.0$	$0$	equivalent — same optimal policy everywhere
$0.5$	$0.5$	fairly similar
$0.0$	$\approx 0.707$	unrelated
$-0.5$	$\approx 0.866$	actively disagreeing (our $R_1$ vs $R_3$)
$-1.0$	$1$	exact opposites

Because it is a scaled Euclidean distance on a sphere, the triangle inequality is inherited for free. It is a pseudometric rather than a metric only because distinct-but-equivalent rewards map to the same point — $d=0$ does not imply $R_1=R_2$. That is the design goal, not a defect.

---

7. The three checks, together

pair	why	$\rho$	$d_{\text{EPIC}}$
$R_1$ vs $R_2=R_1+\gamma \Phi (s^{\prime })-\Phi (s)$	canonical forms identical	$1$	0 ✓
$R_1$ vs $100R_1+7$	$C$ is linear, $\rho$ scale-free	$1$	0 ✓
$R_1$ vs $R_3$ (goal at B)	genuinely different objective	$-0.5$	0.87 ✓

Compare against what $L_2$ would have told you: large, large, and moderate — wrong, wrong, and uninformative.

---

8. The regret bound (why the number means something)

Without a theorem connecting $d_{\text{EPIC}}$ to behaviour, this would just be a heuristic that happens to be invariant. The headline result:

Low EPIC distance implies low regret. If $d_{\text{EPIC}}(R_1,R_2)$ is small, then a policy optimal under $R_2$ has bounded regret when evaluated under $R_1$ — and the bound holds even under different transition dynamics than those used to compute the distance.

That last clause is the interesting one, and it follows from EPIC being deliberately dynamics-independent. It addresses the second defect from §1: a low EPIC distance is evidence about deployment environments you never tested in.

The bound scales with the magnitude of the rewards and with how well $\mathcal{D}$ covers the transitions the relevant policies actually visit.

Flags to check before citing this in the thesis

• Get the exact constants and coverage conditions from §4 of the paper — restated qualitatively here.
• The implication runs one way. Low EPIC ⟹ low regret. High EPIC does not imply high regret; it may just mean the two rewards disagree on transitions nothing ever visits.
• Per the STARC authors, the bound covers only optimal policies, not arbitrary policy pairs. See §10.

---

9. Computing it in practice

def epic_distance(R1, R2, transitions, states, actions, gamma):
    """
    R1, R2      : callables (s, a, s') -> float
    transitions : sampled batch ~ 𝒟          (coverage)
    states      : sampled batch ~ 𝒟_S        (canonicalisation)
    actions     : sampled batch ~ 𝒟_A        (canonicalisation)
    """
    def canonicalise(R):
        # Ψ(s) = E_{A,S'}[R(s, A, S')]
        psi = {s: mean(R(s, a, sp) for a in actions for sp in states)
               for s in set(states) | set(all_states_in(transitions))}
        psi_bar = mean(psi[s] for s in states)
        return lambda s, a, sp: (R(s, a, sp)
                                 + gamma * psi[sp]
                                 - psi[s]
                                 - gamma * psi_bar)
 
    C1, C2 = canonicalise(R1), canonicalise(R2)
    x = np.array([C1(s, a, sp) for (s, a, sp) in transitions])
    y = np.array([C2(s, a, sp) for (s, a, sp) in transitions])
    rho = np.corrcoef(x, y)[0, 1]
    return np.sqrt((1 - rho) / 2)

Cost: $O(|\text{states}|\cdot |\text{actions}|\cdot |\text{states}|)$ reward evaluations to build $\Psi$, plus $O(|\text{transitions}|)$ for the correlation. No policy optimisation, no value iteration, no environment rollouts. This cheapness is the entire practical argument for the method.

Reference implementation: HumanCompatibleAI/evaluating-rewards.

---

10. Where it breaks, and the successors

10.1 The impossible-transition problem

Look again at the canonicalisation: it evaluates $R(S,A,S^{\prime })$ where $S$ and $S^{\prime }$ are drawn independently. In our three-state chain, that includes asking for the reward of going directly from A to C — a transition the dynamics never permit.

EPIC evaluates rewards on all state–state combinations regardless of whether those transitions are physically possible. A learned reward network will happily output something for such inputs — arbitrary extrapolation from a region it was never trained on. Those arbitrary numbers then enter $\Psi$, and hence every canonical value.

In a three-state gridworld you can enumerate everything and it hardly matters. In a high-dimensional continuous space, the fraction of sampled $(s,s^{\prime })$ pairs that are dynamically reachable can approach zero.

This is a deliberate trade, not an oversight: independence from dynamics is what buys the cross-dynamics regret bound in §8. You cannot have both.

10.2 Sensitivity to $\mathcal{D}$

Both the canonicalisation distributions and the coverage distribution are free choices, and the number moves when you move them. EPIC is more robust to this than the baselines it was compared against — but “more robust” is not “insensitive”. Report your distributions alongside any EPIC number, or the number is not reproducible.

10.3 Successors

method	fix	trade-off
DARD (Wulfe et al., ICLR 2022) — Dynamics-Aware Reward Distance	canonicalises using the actual transition dynamics, so rewards are only evaluated on reachable transitions	tighter when dynamics are known; no regret bound derived, and loses the cross-dynamics guarantee
STARC (Skalse et al., ICLR 2024)	generalises the canonicalise → normalise → compare recipe into a whole family; bounds in both directions	also dynamics-dependent; more machinery

STARC’s specific criticisms of EPIC, worth knowing before you cite EPIC as the standard: the regret bound is weaker and less general, it covers only optimal policies rather than arbitrary policy pairs, and they argue EPIC’s notion of regret does not capture what practitioners actually care about. Appendix A of the STARC paper is an extended analysis of EPIC and is probably the best critical reading available.

---

11. Relevance to the thesis

Direct, on two fronts — see thesis-hub and continuous-action-extension-directions.

Why it fits Bayesian IRL particularly well. The invariances EPIC quotients out are exactly the directions the BIRL likelihood cannot see:

• Shaping is unidentifiable from demonstrations by construction. The expert behaves identically under $R_1$ and $R_2$ from §2.3, so no quantity of data separates them. A metric that penalises this is penalising the method for failing at something provably impossible.
• Scale is entangled with the rationality parameter. In the Boltzmann model $\pi (a|s)\propto \exp (\beta Q(s,a))$, a large reward with small $\beta$ is observationally identical to a small reward with large $\beta$. See ramachandran-2007-bayesian-irl. EPIC’s affine invariance handles this directly.

Why it fits Bayesian IRL, not just IRL. Because there is no policy optimisation, EPIC is cheap enough to evaluate per posterior sample. Instead of one number for a point estimate, you get a distribution of EPIC distances across the ValueWalk / MCMC posterior. That is the right object for calibration claims — you can ask whether posterior spread in EPIC space tracks actual error, which point-estimate metrics simply cannot express.

The continuous-action complication. Both expectations defining $\Psi$ become integrals requiring Monte-Carlo approximation:

$\Psi (s)=\int \int R(s,a,s^{\prime })d{\mathcal{D}}_A(a)d{\mathcal{D}}_S(s^{\prime })$

• ${\mathcal{D}}_A$ over an unbounded or high-dimensional continuous action space is a genuine modelling decision, not a uniform-over-$k$-actions default. Uniform over a box? Gaussian around the expert’s action? Under the learned policy?
• §10.1 bites much harder here: the impossible-transition fraction is far worse in continuous state spaces than in a 3-state chain.
• Monte-Carlo error in $\Psi$ propagates into every canonical value. Estimator variance needs a budget of its own.
• This may make DARD the better default for the continuous experiments — worth an explicit decision rather than defaulting to EPIC because it is better known.

Relevant neighbours for baselines: AVRIL, IQ-Learn, DistIRL.

---

12. Exercises

1. Telescoping. Verify by hand that the shaped total for path A→B→A→B→C equals 11, matching the direct-route total. Now change $\Phi (B)$ to $-3$ and confirm both totals are still equal to each other.
2. Canonicalisation is idempotent. Show $C(C(R))=C(R)$. (Hint: what is $\Psi$ of an already-canonical reward?)
3. Constant rewards. What is $C(R)$ when $R\equiv c$? What does that imply about $d_{\text{EPIC}}$ between two constant rewards, and why is that degenerate case unavoidable?
4. Sensitivity. Recompute the $R_1$ vs $R_3$ distance with $\mathcal{D}$ restricted to only the transitions the shortest path A→B→C visits. Does the number change? What does that tell you about §10.2?
5. Negative scaling. EPIC is invariant to $\lambda >0$. Construct a two-state example showing why $\lambda <0$ must not be treated as equivalent.
6. Continuous-action estimator. For a 1-D action space $a\in [-1,1]$ and $R(s,a,s^{\prime })=-a^2+1[s^{\prime }=g]$, write $\Psi (s)$ in closed form under ${\mathcal{D}}_A=\text{Uniform}[-1,1]$. Then compare against a 10-sample Monte-Carlo estimate and characterise the variance.

---

13. References

Primary

• Gleave, Dennis, Legg, Russell & Leike (2021). Quantifying Differences in Reward Functions. ICLR 2021. arXiv:2006.13900
• Code: github.com/HumanCompatibleAI/evaluating-rewards

Foundational

• Ng, Harada & Russell (1999). Policy Invariance Under Reward Transformations. ICML. — origin of potential shaping and the proof that it preserves optimal policies.

Successors

• Wulfe et al. (2022). Dynamics-Aware Comparison of Learned Reward Functions (DARD). ICLR 2022. arXiv:2201.10081
• Skalse et al. (2024). STARC: A General Framework For Quantifying Differences Between Reward Functions. ICLR 2024. arXiv:2309.15257

---

Open questions for Ondřej (advisor)

• Is EPIC the metric the QVIRL/ValueWalk line already reports, or does the group use something else for reward recovery?
• For the continuous-action extension: EPIC or DARD as the primary metric? The impossible-transition issue (§10.1) seems considerably worse in continuous state spaces, but DARD gives up the regret bound.
• Is per-posterior-sample EPIC (§11) something that has been done in the BIRL literature, or would reporting the distance distribution be a small novel contribution in itself?
• How much does the group weight the STARC critique — is EPIC still the community default in 2026, or has the field moved?