Motivated by the trade-off between exploitation and exploration in reinforcement learning, we study a continuous-time entropy-regularized mean variance portfolio selection problem in the presence of jumps. We propose an exploratory SDE for the wealth process associated with multiple risky assets which exhibit Lévy jumps. In contrast to the existing literature, we study the limiting behavior of the natural discrete-time formulation of the wealth process associated to a randomized control in order to derive the continuous-time dynamics. We then show that an optimal distributional control of the continuous-time entropy-regularized exploratory mean-variance problem is Gaussian. The respective optimal wealth process solves a linear SDE whose representation is explicitly obtained.