In this paper, we investigate certain combinatorial numbers, the \textit{moment generating Stirling numbers}. They are a special case of Hsu's generalized Stirling numbers and satisfy many more properties and combinatorial identities than are known in the general case. As application, we provide the computation of the moments and central moments of the phase type distribution, the recurrence time in Markov chains, the geometric distribution, the negative binomial distribution and of a class of distributions generalizing the negative binomial distribution. All computations can be performed in closed form without recursion. We also present the relationship to the Markov renewal process.