Rerandomization enforces covariate balance across treatment groups in the design stage of experiments. Despite its intuitive appeal, its theoretical justification remains unsatisfying because its benefits of improving efficiency for estimating the average treatment effect diminish if we use regression adjustment in the analysis stage. To strengthen the theory of rerandomization, we show that it mitigates false discoveries resulting from $p$-hacking, the practice of strategically selecting covariates to get more significant $p$-values. Moreover, we show that rerandomization with a sufficiently stringent threshold can resolve $p$-hacking. As a byproduct, our theory offers guidance for choosing the threshold in rerandomization in practice.