For every graph $X$, we consider the class of all connected $\{K_{1,3}, X\}$-free graphs which are distinct from an odd cycle and have independence number at least $4$, and we show that all graphs in the class are perfect if and only if $X$ is an induced subgraph of some of $P_6$, $K_1 \cup P_5$, $2P_3$, $Z_2$ or $K_1 \cup Z_1$. Furthermore, for $X$ chosen as $2K_1 \cup K_3$, we list all eight imperfect graphs belonging to the class; and for every other choice of $X$, we show that there are infinitely many such graphs. In addition, for $X$ chosen as $B_{1,2}$, we describe the structure of all imperfect graphs in the class.