The (\textsc{Weighted}) \textsc{Subset Feedback Vertex Set} problem is a generalization of the classical \textsc{Feedback Vertex Set} problem and asks for a vertex set of minimum (weighted) size that intersects all cycles containing a vertex of a predescribed set of vertices. Although the two problems exhibit different computational complexity on split graphs, no similar characterization is known on other classes of graphs. Towards the understanding of the complexity difference between the two problems, it is natural to study the importance of a structural graph parameter. Here we consider graphs of bounded independent set number for which it is known that \textsc{Weighted Feedback Vertex Set} is solved in polynomial time. We provide a dichotomy result with respect to the size of a maximum independent set. In particular we show that \textsc{Weighted Subset Feedback Vertex Set} can be solved in polynomial time for graphs of independent set number at most three, whereas we prove that the problem remains NP-hard for graphs of independent set number four. Moreover, we show that the (unweighted) \textsc{Subset Feedback Vertex Set} problem can be solved in polynomial time on graphs of bounded independent set number by giving an algorithm with running time $n^{O(d)}$, where $d$ is the size of a maximum independent set of the input graph. To complement our results, we demonstrate how our ideas can be extended to other terminal set problems on graphs of bounded independent set size. Based on our findings for \textsc{Subset Feedback Vertex Set}, we settle the complexity of \textsc{Node Multiway Cut}, a terminal set problem that asks for a vertex set of minimum size that intersects all paths connecting any two terminals, as well as its variants where nodes are weighted and/or the terminals are deletable, for every value of the given independent set number.