We revisit the topic of polynomial kernels for Vertex Cover relative to structural parameters. Our starting point is a recent paper due to Fomin and Strømme [WG 2016] who gave a kernel with $\mathcal{O}(|X|^{12})$ vertices when $X$ is a vertex set such that each connected component of $G-X$ contains at most one cycle, i.e., $X$ is a modulator to a pseudoforest. We strongly generalize this result by using modulators to $d$-quasi-forests, i.e., graphs where each connected component has a feedback vertex set of size at most $d$, and obtain kernels with $\mathcal{O}(|X|^{3d+9})$ vertices. Our result relies on proving that minimal blocking sets in a $d$-quasi-forest have size at most $d+2$. This bound is tight and there is a related lower bound of $\mathcal{O}(|X|^{d+2-ε})$ on the bit size of kernels. In fact, we also get bounds for minimal blocking sets of more general graph classes: For $d$-quasi-bipartite graphs, where each connected component can be made bipartite by deleting at most $d$ vertices, we get the same tight bound of $d+2$ vertices. For graphs whose connected components each have a vertex cover of cost at most $d$ more than the best fractional vertex cover, which we call $d$-quasi-integral, we show that minimal blocking sets have size at most $2d+2$, which is also tight. Combined with existing randomized polynomial kernelizations this leads to randomized polynomial kernelizations for modulators to $d$-quasi-bipartite and $d$-quasi-integral graphs. There are lower bounds of $\mathcal{O}(|X|^{d+2-ε})$ and $\mathcal{O}(|X|^{2d+2-ε})$ for the bit size of such kernels.