Abstract:We study model-free Q-learning in finite-horizon episodic Markov Decision Processes (MDPs) with stationary dynamics across episodes. We identify a central issue in nascent model-free posterior-sampling works: the reliance on delayed learning in order to prove theoretical guarantees. In particular, we identify three opportunities for faster learning - (i) value-function update order, (ii) update frequencies, and (iii) value-function initialization. Using Wang et al.'s RandomizedQ as a basis, we illustrate these changes and their individual (as well as cumulative) impact in multiple empirical studies. We find that our combined modifications, termed ReversedQ, improve scaled mean cumulative reward compared to RandomizedQ, from 9.53% to 78.78% in the Bidirectional Diabolical Combination Lock (BDCL), and from 21.76% to 61.81% in a chain MDP.
| Comments: | This paper contains 5 pages and 2 figures. To be presented at the Adaptive and Learning Agents workshop (ALA 2026) at AAMAS 2026 |
| Subjects: | Machine Learning (cs.LG) |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2605.20592 [cs.LG] |
| (or arXiv:2605.20592v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20592 arXiv-issued DOI via DataCite (pending registration) |
From: Sofia Miskala-Dinc [view email]
[v1]
Wed, 20 May 2026 00:58:07 UTC (1,109 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。