In this paper, we show that for a non-trivial quasi-symmetric $2$-design $\mathcal{D}$ with two block intersection numbers $x=0$ and $2\leq y\leq10$, if $G\leq \mathrm{Aut}(\mathcal{D})$ is flag-transitive and point-primitive, then $G$ is either of affine type or almost simple type. Moreover, we prove that the socle of $G$ cannot be an alternating group. If the socle of $G$ is a sporadic group, then $\mathcal{D}$ and $G$ must be one of the following: $\mathcal{D}$ is a $2$-$(12,6,5)$ design with block intersection numbers $0,3$ and $G=\mathrm{M}_{11}$, or $\mathcal{D}$ is a $2$-$(22,6,5)$ design with block intersection numbers $0,2$ and $G=\mathrm{M}_{22}$ or $\mathrm{M}_{22}:2$.