Abstract:Inverse reinforcement learning (IRL) recovers the loss function of a forward learner from its observed responses. Adaptive IRL aims to reconstruct the loss function of a forward learner by passively observing its gradients as it performs reinforcement learning (RL). This paper proposes a novel passive Langevin-based algorithm that achieves adaptive IRL. The key difficulty in adaptive IRL is that the required gradients in the passive algorithm are counterfactual, that is, they are conditioned on events of probability zero under the forward learner's trajectory. Therefore, naive Monte Carlo estimators are prohibitively inefficient, and kernel smoothing, though common, suffers from slow convergence. We overcome this by employing Malliavin calculus to efficiently estimate the required counterfactual gradients. We reformulate the counterfactual conditioning as a ratio of unconditioned expectations involving Malliavin quantities, thus recovering standard estimation rates. We derive the necessary Malliavin derivatives and their adjoint Skorohod integral formulations for a general Langevin structure, and provide a concrete algorithmic approach which exploits these for counterfactual gradient estimation.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.01345 [cs.LG] |
| (or arXiv:2604.01345v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.01345 arXiv-issued DOI via DataCite |
From: Luke Snow [view email]
[v1]
Wed, 1 Apr 2026 19:56:02 UTC (1,131 KB)
[v2]
Wed, 6 May 2026 16:05:17 UTC (1,131 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。