Let $I(G)$ be the edge ideal of a simple graph $G$ over a field k. We prove that $${\rm reg}(\overline {I(G)^s}) = {\rm reg}(I(G)^s),$$ for all $s \le 4$. Furthermore, we provide an example of a graph $G$ such that $${\rm reg} I(G)^s = {\rm reg} \overline{I(G)^s} = {\rm reg} I(G)^{(s)} = \begin{cases} 5 + 2s & \text{ if char k} = 2 \\ 4 + 2s & \text{ if char k} \neq 2, \end{cases}$$ for all $s \ge 1.$