We consider the rough differential equation with drift driven by a Gaussian geometric rough path. Under natural conditions on the rough path, namely non-determinism, and uniform ellipticity conditions on the diffusion coefficient, we prove path-by-path well-posedness of the equation for poorly regular drifts. In the case of the fractional Brownian motion $B^H$ for $H>\frac14$, we prove that the drift may be taken to be $κ>0$ Hölder continuous and bounded for $κ>\frac32 - \frac1{2H}$. A flow transform of the equation and Malliavin calculus for Gaussian rough paths are used to achieve such a result.