The minimum forcing number of a graph $G$ is the smallest number of edges simultaneously contained in a unique perfect matching of $G$. Zhang, Ye and Shiu \cite{HDW} showed that the minimum forcing number of any fullerene graph was bounded below by $3$. However, we find that there exists exactly one excepted fullerene $F_{24}$ with the minimum forcing number $2$. In this paper, we characterize all fullerenes with the minimum forcing number $3$ by a construction approach. This also solves an open problem proposed by Zhang et al. We also find that except for $F_{24}$, all fullerenes with anti-forcing number $4$ have the minimum forcing number $3$. In particular, the nanotube fullerenes of type $(4, 2)$ are such fullerenes.