In this article, we prove that a small random perturbation of dynamical system with multiple stable equilibria converges to a Markov chain whose states are neighborhoods of the deepest stable equilibria, under a suitable time-rescaling, provided that the perturbed dynamics is reversible in time. Such a result has been anticipated from 1970s, when the foundation of mathematical treatment for this problem has been established by Freidlin and Wentzell. We solve this long-standing problem by reducing the entire analysis to an investigation of the solution of an associated Poisson equation, and furthermore provide a method to carry out this analysis by using well-known test functions in a novel manner.