For graphs $G$ and $H$, an $H$-coloring of $G$ is an edge-preserving mapping from $V(G)$ to $V(H)$. Note that if $H$ is the triangle, then $H$-colorings are equivalent to $3$-colorings. In this paper we are interested in the case that $H$ is the five-vertex cycle $C_5$. A minimal obstruction to $C_5$-coloring is a graph that does not have a $C_5$-coloring, but every proper induced subgraph thereof has a $C_5$-coloring. In this paper we are interested in minimal obstructions to $C_5$-coloring in $F$-free graphs, i.e., graphs that exclude some fixed graph $F$ as an induced subgraph. Let $P_t$ denote the path on $t$ vertices, and let $S_{a,b,c}$ denote the graph obtained from paths $P_{a+1},P_{b+1},P_{c+1}$ by identifying one of their endvertices. We show that there is only a finite number of minimal obstructions to $C_5$-coloring among $F$-free graphs, where $F \in \{ P_8, S_{2,2,1}, S_{3,1,1}\}$ and explicitly determine all such obstructions. This extends the results of Kamiński and Pstrucha [Discr. Appl. Math. 261, 2019] who proved that there is only a finite number of $P_7$-free minimal obstructions to $C_5$-coloring, and of Dębski et al. [ISAAC 2022 Proc.] who showed that the triangle is the unique $S_{2,1,1}$-free minimal obstruction to $C_5$-coloring. We complement our results with a construction of an infinite family of minimal obstructions to $C_5$-coloring, which are simultaneously $P_{13}$-free and $S_{2,2,2}$-free. We also discuss infinite families of $F$-free minimal obstructions to $H$-coloring for other graphs $H$.