This paper is devoted to the study of a stochastic process obtained by random switching between a finite collection of vector fields. Such processes have recently been the focus of much attention in the case where the switching times are exponentially distributed, i.e., Markovian switching. In this contribution, we admit any distribution on $\mathbb{R}_+$ as a law for the switching times. We show that whenever this law is not singular with respect to the Lebesgue measure, the stochastic process obtained from the random switching is Feller. More importantly, we give conditions on the switching and on the vector fields ensuring that the Lie bracket condition considered in the Markovian case in Bakhtin and Hurth (2012) and Benaïm, Le Borgne, Malrieu and Zitt (2015) still imply ergodicity of the process.