A graph $G$ is $k$-vertex-critical if $χ(G)=k$ but $χ(G-v)<k$ for all $v\in V(G)$ where $χ(G)$ denotes the chromatic number of $G$. We show that there are only finitely many $k$-critical $(P_3+\ell P_1)$-free graphs for all $k$ and all $\ell$. Together with previous results, the only graphs $H$ for which it is unknown if there are an infinite number of $k$-vertex-critical $H$-free graphs is $H=(P_4+\ell P_1)$ for all $\ell\ge 1$. We consider a restriction on the smallest open case, and show that there are only finitely many $k$-vertex-critical (gem, co-gem)-free graphs for all $k$, where gem$=\overline{P_4+P_1}$. To do this, we show the stronger result that every vertex-critical (gem, co-gem)-free graph is either complete or a clique expansion of $C_5$. This characterization allows us to give the complete list of all $k$-vertex-critical (gem, co-gem)-free graphs for all $k\le 16$