The paper introduces a novel, hierarchical preconditioner based on nested dissection and hierarchical matrix compression. The preconditioner is intended for continuous and discontinuous Galerkin formulations of elliptic problems. We exploit the property that Schur complements arising in such problems can be well approximated by hierarchical matrices. An approximate factorization can be computed matrix-free and in a (quasi-)linear number of operations. The nested dissection is specifically designed to aid the factorization process using hierarchical matrices. We demonstrate the viability of the preconditioner on a range of 2D problems, including the Helmholtz equation and the elastic wave equation. Throughout all tests, including wave phenomena with high wavenumbers, the generalized minimal residual method (GMRES) with the proposed preconditioner converges in a very low number of iterations. We demonstrate that this is due to the hierarchical nature of our approach which makes the high wavenumber limit manageable.