Non-linear Hawkes processes with memory kernels given by the sum of Erlang kernels are considered. It is shown that their stability properties can be studied in terms of an associated class of piecewise deterministic Markov processes, called Markovian cascades of successive memory terms. Explicit conditions implying the positive Harris recurrence of these processes are presented. The proof is based on integration by parts with respect to the jump times. A crucial property is the non-degeneracy of the transition semigroup which is obtained thanks to the invertibility of an associated Vandermonde matrix. For Lipschitz continuous rate functions we also show that these Markovian cascades converge to equilibrium exponentially fast with respect to the Wasserstein distance. Finally, an extension of the classical thinning algorithm is proposed to simulate such Markovian cascades.