Let $h(G)$ denote the largest $t$ such that $G$ contains $K_t$ as a minor and $χ(G)$ be the chromatic number of $G$ respectively. In 1943, Hadwiger conjectured that $h(G) \geq χ(G)$ for any graph $G$. In this paper, we prove that Hadwiger's conjecture holds for $H$-free graphs with independence number two, where $H$ is one of some specified graphs.