


























Abstract:Multi-objective reinforcement learning (MORL) algorithms extend conventional reinforcement learning (RL) to the more general case of problems with multiple, conflicting objectives, represented by vector-valued rewards. Widely-used scalar RL methods such as Q-learning can be modified to handle multiple objectives by (1) learning vector-valued value functions, and (2) performing action selection using a scalarisation or ordering operator which reflects the user's preferences with respect to the different objectives. This paper investigates two previously unreported issues which can hinder the performance of value-based MORL algorithms when applied in conjunction with a non-linear utility function -- value function interference, and sensitivity to overestimation. We illustrate the nature of these phenomena on simple multi-objective MDPs using a tabular implementation of multiobjective Q-learning.
| Comments: | This updates our previous pre-print to add extended discussion of value-function interference as well as new material illustrating the interaction between Q-value overestimation and non-linear utility |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2402.06266 [cs.LG] |
| (or arXiv:2402.06266v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2402.06266 arXiv-issued DOI via DataCite |
From: Peter Vamplew [view email]
[v1]
Fri, 9 Feb 2024 09:28:01 UTC (302 KB)
[v2]
Wed, 22 Apr 2026 05:23:45 UTC (1,455 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。