Physical environment understanding is vital in delivering immersive and interactive mobile augmented reality (AR) user experiences. Recently, we have witnessed a transition in the design of environment understanding systems, from visual data focused to centering on the concept of spatial context, including user, device, and environment information. Even though spatial context can benefit the environment understanding tasks, e.g., we demonstrate in a case study that lighting estimation performance can be improved by as much as 59%, not all environment understanding systems support spatial context. Furthermore, even for the environment understanding systems that support spatial context, not all useful spatial context has been leveraged; and the design and implementation can differ vastly. In this paper, we advocate for the design of a spatial context-aware and shared environment understanding system that can effectively support multiple tasks simultaneously. Fundamentally, there are many practical challenges in designing such a unified environment understanding system. We discuss the challenges in the context of three open questions, (i) how to fully leverage user mobility, (ii) how to design a unified data model, and (iii) finally how to build a shared system for multiple tasks.