We study a stochastic system of interacting neurons and its metastable properties. The system consists of $N$ neurons, each spiking randomly with rate depending on its membrane potential. At its spiking time, the neuron potential is reset to $0$ and all other neurons receive an additional amount $h/N$ of potential. In between successive spike times, each neuron looses potential at exponential speed. We study this system in the supercritical regime, that is, for sufficiently high values of the synaptic weight $h.$ Under very mild conditions on the spiking rate function, is has been shown in Duarte and Ost \cite{do} that the only invariant distribution of the finite system is the trivial measure $ δ_{\bf 0}$ corresponding to extinction of the process. Under minimal conditions on the behavior of the spiking rate function in the vicinity of $0$, we prove that the extinction time arrives at exponentially late times in $ N$, and discuss the stability of the equilibrium $δ_{\bf 0}$ for the non-linear mean-field limit process depending on the parameters of the dynamics. We then specify our study to the case of saturating spiking rates and show that, under suitable conditions on the parameters of the model, 1) the non-linear mean-field limit admits a unique and globally attracting equilibrium and 2) the rescaled exit times for the mean spiking rate of a finite system from a neighbourhood of the non-linear equilibrium rate converge in law to an exponential distribution, as the system size diverges. In other words, the system exhibits a metastable behavior.