Abstract:Reinforcement Learning from Human Feedback (RLHF) can reveal implicit objectives such as safety considerations that go beyond task completion. In this work, we focus on the common safety criteria embedded in crowd preference datasets, where different users may express distinct preferences or objectives, yet follow similar safety principles. Our aim is to discover shared safety criteria from crowd preferences and then transfer them to downstream RL tasks to regularize agent behavior and enforce safety. We first show that direct reward combination-optimizing a preference-learned reward model together with downstream task rewards-has inherent limitations. Motivated by this, we propose Safe Crowd Preference-based RL, a hierarchical framework that extracts safety-aligned skills from crowd preferences and composes them via a high-level policy to safely solve downstream tasks. Experiments across safe RL environments and a preliminary LLM-style task with diverse user goals and shared safety constraints demonstrate that our approach substantially lowers safety costs without access to explicit safety rewards, while achieving task performance comparable to oracle methods trained with ground-truth safety signals.
| Comments: | Accepted to ICML 2026. Conference paper |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.21822 [cs.AI] |
| (or arXiv:2605.21822v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21822 arXiv-issued DOI via DataCite (pending registration) |
From: Qian Lin [view email]
[v1]
Wed, 20 May 2026 23:44:06 UTC (802 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。