Given a vertex-weighted graph $G=(V,E)$ and a set $S \subseteq V$, a subset feedback vertex set $X$ is a set of the vertices of $G$ such that the graph induced by $V \setminus X$ has no cycle containing a vertex of $S$. The \textsc{Subset Feedback Vertex Set} problem takes as input $G$ and $S$ and asks for the subset feedback vertex set of minimum total weight. In contrast to the classical \textsc{Feedback Vertex Set} problem which is obtained from the \textsc{Subset Feedback Vertex Set} problem for $S=V$, restricted to graph classes the \textsc{Subset Feedback Vertex Set} problem is known to be NP-complete on split graphs and, consequently, on chordal graphs. However as \textsc{Feedback Vertex Set} is polynomially solvable for AT-free graphs, no such result is known for the \textsc{Subset Feedback Vertex Set} problem on any subclass of AT-free graphs. Here we give the first polynomial-time algorithms for the problem on two unrelated subclasses of AT-free graphs: interval graphs and permutation graphs. As a byproduct we show that there exists a polynomial-time algorithm for circular-arc graphs by suitably applying our algorithm for interval graphs. Moreover towards the unknown complexity of the problem for AT-free graphs, we give a polynomial-time algorithm for co-bipartite graphs. Thus we contribute to the first positive results of the \textsc{Subset Feedback Vertex Set} problem when restricted to graph classes for which \textsc{Feedback Vertex Set} is solved in polynomial time.