Beamforming in multiple input multiple output (MIMO) systems is one of the key technologies for modern wireless communication. Creating appropriate sector-specific broadcast beams are essential for enhancing the coverage of cellular network and for improving the broadcast operation for control signals. However, in order to maximize the coverage, patterns for broadcast beams need to be adapted based on the users' distribution and movement over time. In this work, we present self-tuning sectorization: a deep reinforcement learning framework to optimize MIMO broadcast beams autonomously and dynamically based on user' distribution in the network. Taking directly UE measurement results as input, deep reinforcement learning agent can track and predict the UE distribution pattern and come up with the best broadcast beams for each cell. Extensive simulation results show that the introduced framework can achieve the optimal coverage, and converge to the oracle solution for both single sector and multiple sectors environment, and for both periodic and Markov mobility patterns.