Internet of things, supported by machine-to-machine (M2M) communications, is one of the most important applications for future 6th generation (6G) systems. A major challenge facing by 6G is enabling a massive number of M2M devices to access networks in a timely manner. Therefore, this paper exploits the spatial selectivity of massive multi-input multi-output (MIMO) to reduce the collision issue when massive M2M devices initiate random access simultaneously. In particular, a beam-based random access protocol is first proposed to make efficient use of the limited uplink resources for massive M2M devices. To address the non-uniform distribution of M2M devices in the space and time dimensions, an Markov decision process (MDP) problem with the objective of minimizing the average access delay is then formulated. Next, we present a dynamic beam-based access scheme based on the double deep Q network (DDQN) algorithm to solve the optimal policy. Finally, simulations are conducted to demonstrate the effectiveness of the proposed scheme including the model training and random access performance.