The millimeter-wave (mmWave) communication technology, which employs large-scale antenna arrays, enables inherent sensing capabilities. Simultaneous localization and mapping (SLAM) can utilize channel multipath angle estimates to realize integrated sensing and communication design in 6G communication systems. However, existing works have ignored the significant overhead required by the mmWave beam management when implementing SLAM with angle estimates. This study proposes a joint beam management and SLAM design that utilizes the strong coupling between the radio map and channel multipath for simultaneous beam management, localization, and mapping. In this approach, we first propose a hierarchical sweeping and sensing service design. The path angles are estimated in the hierarchical sweeping, enabling angle-based SLAM with the aid of an inertial measurement unit (IMU) to realize sensing service. Then, feature-aided tracking is proposed that utilizes prior angle information generated from the radio map and IMU. Finally, a switching module is introduced to enable flexible switching between hierarchical sweeping and feature-aided tracking. Simulations show that the proposed joint design can achieve sub-meter level localization and mapping accuracy (with an error < 0.5 m). Moreover, the beam management overhead can be reduced by approximately 40% in different wireless environments.