This note lays part of the theoretical ground for a definition of differential systems modeling reinforcement learning in continuous time non-Markovian rough environments. Specifically we focus on optimal relaxed control of rough equations (the term relaxed referring to the fact that controls have to be considered as measure valued objects). With reinforcement learning in view, our reward functions encompass forms that involve an entropy-type term favoring exploration. In this context, our contribution focuses on a careful definition of the corresponding relaxed Hamilton-Jacobi-Bellman (HJB)-type equation. A substantial part of our endeavor consists in a precise definition of the notion of test function and viscosity solution for the rough relaxed PDE obtained in this framework. Note that this task is often merely sketched in the rough viscosity literature, in spite of the fact that it gives a proper meaning to the differential system at stake. In the last part of the paper we prove that the natural value function in our context solves a relaxed rough HJB equation in the viscosity sense.