In this paper, energy efficient power allocation for the uplink of a multi-cell massive MIMO system is investigated. With the simplified power consumption model, the problem of power allocation is formulated as a constrained Markov decision process (CMDP) framework with infinite-horizon expected discounted total reward, which takes into account different quality of service (QoS) requirements for each user terminal (UT). We propose an offline solution containing the value iteration and Q-learning algorithms, which can obtain the global optimum power allocation policy. Simulation results show that our proposed policy performs very close to the ergodic optimal policy.