We study the $\mathcal{F}$-center problem with outliers: given a metric space $(X,d)$, a general down-closed family $\mathcal{F}$ of subsets of $X$, and a parameter $m$, we need to locate a subset $S\in \mathcal{F}$ of centers such that the maximum distance among the closest $m$ points in $X$ to $S$ is minimized. Our main result is a dichotomy theorem. Colloquially, we prove that there is an efficient $3$-approximation for the $\mathcal{F}$-center problem with outliers if and only if we can efficiently optimize a poly-bounded linear function over $\mathcal{F}$ subject to a partition constraint. One concrete upshot of our result is a polynomial time $3$-approximation for the knapsack center problem with outliers for which no (true) approximation algorithm was known.