The computation of free energies is a common issue in statistical physics. A natural technique to compute such high dimensional integrals is to resort to Monte Carlo simulations. However these techniques generally suffer from a high variance in the low temperature regime, because the expectation is often dominated by high values corresponding to rare system trajectories. A standard way to reduce the variance of the estimator is to modify the drift of the dynamics with a control enhancing the probability of rare event, leading to so-called importance sampling estimators. In theory, the optimal control leads to a zero-variance estimator; it is however defined implicitly and computing it is of the same difficulty as the original problem. We propose here a general strategy to build approximate optimal controls in the small temperature limit for diffusion processes, with the first goal to reduce the variance of free energy Monte Carlo estimators. Our construction builds upon low noise asymptotics by expanding the optimal control around the instanton, which is the path describing most likely fluctuations at low temperature. This technique not only helps reducing variance, but it is also interesting as a theoretical tool since it differs from usual small temperature expansions (WKB ansatz). As a complementary consequence of our expansion, we provide a perturbative formula for computing the free energy in the small temperature regime, which refines the now standard Freidlin--Wentzell asymptotics. We compute this expansion explicitly for lower orders, and explain how our strategy can be extended to an arbitrary order of accuracy. We support our findings with illustrative numerical examples.