Virtual and augmented reality are currently enjoying a great deal of attention from the research community and the industry towards their adoption within industrial spaces and processes. However, the current design and implementation landscape is still very fluid, while the community as a whole has not yet consolidated into concrete design directions, other than basic patterns. Other open issues include the choice over a cloud or edge-based architecture when designing such systems. Within this work, we present our approach for a monitoring intervention inside a factory space utilizing both VR and AR, based primarily on edge computing, while also utilizing the cloud. We discuss its main design directions, as well as a basic ontology to aid in simple description of factory assets. In order to highlight the design aspects of our approach, we present a prototype implementation, based on a use case scenario in a factory site, within the context of the ENERMAN H2020 project.