Weak solutions to the homogeneous Boltzmann equation with increasing energy have been constructed by Lu and Wennberg. We consider an underlying microscopic stochastic model with binary collision (Kac's model) and show that these solutions are atypical. More precisely, we prove that the probability of observing these paths is exponentially small in the number of particles and compute the exponential rate. This result is obtained by improving the established large deviation estimates in the canonical setting. Key ingredients are the extension of Sanov's theorem to the microcanonical ensemble and large deviations for the Kac's model in the microcanonical setting.