Robotics has been a popular field of research in the past few decades, with much success in industrial applications such as manufacturing and logistics. This success is led by clearly defined use cases and controlled operating environments. However, robotics has yet to make a large impact in domestic settings. This is due in part to the difficulty and complexity of designing mass-manufactured robots that can succeed in the variety of homes and environments that humans live in and that can operate safely in close proximity to humans. This paper explores the use of contextual affordances to enable safe exploration and learning in robotic scenarios targeted in the home. In particular, we propose a simple state representation that allows us to extend contextual affordances to larger state spaces and showcase how affordances can improve the success and convergence rate of a reinforcement learning algorithm in simulation. Our results suggest that after further iterations, it is possible to consider the implementation of this approach in a real robot manipulator. Furthermore, in the long term, this work could be the foundation for future explorations of human-robot interactions in complex domestic environments. This could be possible once state-of-the-art robot manipulators achieve the required level of dexterity for the described affordances in this paper.