In this article, we consider the following stochastic fractional diffusion equation \begin{equation*} \left(\partial^β+\dfracν{2}\left(-Δ\right)^{α/ 2}\right) u(t, x)= λ\: I_{0_+}^γ\left[u(t, x) \dot{W}(t, x)\right] ,\quad t>0,\: x \in \mathbb{R}, \end{equation*} where $α>0$, $β\in(0,2]$, $γ\ge 0$, $λ\neq0$, $ν>0$, and $\dot{W}$ is a Gaussian noise which is white or fractional in time and rough in space. We prove the existence and uniqueness of the solution in the Itô-Skorohod sense and obtain the lower and upper bounds for the $p$-th moment. The Hölder regularity of the solution is also studied.