We consider solutions of the linear heat equation in $\mathbb{R}^N$ with isolated singularities. It is assumed that the position of a singular point depends on time and is Hölder continuous with the exponent $α\in (0,1)$. We show that any isolated singularity is removable if it is weaker than a certain order depending on $α$. We also show the optimality of the removability condition by showing the existence of a solution with a nonremovable singularity. These results are applied to the case where the singular point behaves like a fractional Brownian motion with the Hurst exponent $H \in (0,1/2] $. It turns out that $H=1/N$ is critical.