The $H$-Free Edge Deletion problem asks, for a given graph $G$ and an integer $k$, whether it is possible to delete at most $k$ edges from $G$ to make it $H$-free, that is, not containing $H$ as an induced subgraph. The $H$-Free Edge Completion problem is defined similarly, but we add edges instead of deleting them. The study of these two problem families has recently been the subject of intensive studies from the point of view of parameterized complexity and kernelization. In particular, it was shown that the problems do not admit polynomial kernels (under plausible complexity assumptions) for almost all graphs $H$, with several important exceptions occurring when the class of $H$-free graphs exhibits some structural properties. In this work we complement the parameterized study of edge modification problems to $H$-free graphs by considering their approximability. We prove that whenever $H$ is $3$-connected and has at least two non-edges, then both $H$-Free Edge Deletion and $H$-Free Edge Completion are very hard to approximate: they do not admit $\mathrm{poly}(\mathsf{OPT})$-approximation in polynomial time, unless $\mathrm{P}=\mathrm{NP}$, or even in time subexponential in $\mathsf{OPT}$, unless the Exponential Time Hypothesis fails. The assumption of the existence of two non-edges appears to be important: we show that whenever $H$ is a complete graph without one edge, then $H$-Free Edge Deletion is tightly connected to the Min Horn problem, whose approximability is still open. Finally, in an attempt to extend our hardness results beyond $3$-connected graphs, we consider the cases of $H$ being a path or a cycle, and we achieve an almost complete dichotomy there.