Double Hurwitz numbers enumerate branched covers of $\mathbb{CP}^1$ with prescribed ramification over two points and simple ramification elsewhere. In contrast to the single case, their underlying geometry is not well understood. In previous work by the second- and third-named authors, the double Hurwitz numbers were conjectured to satisfy a polynomiality structure and to be governed by the topological recursion, analogous to existing results concerning single Hurwitz numbers. In this paper, we resolve these conjectures by a careful analysis of the semi-infinite wedge representation for double Hurwitz numbers, by pushing further methods previously used for other Hurwitz problems. We deduce a preliminary version of an ELSV-like formula for double Hurwitz numbers, by deforming the Johnson-Pandharipande-Tseng formula for orbifold Hurwitz numbers and using properties of the topological recursion under variation of spectral curves. In the course of this analysis, we unveil certain vanishing properties of the Chiodo classes.