Tverberg's theorem bounds the number of points $\mathbb{R}^d$ needed for the existence of a partition into $r$ parts whose convex hulls intersect. If the points are colored with $N$ colors, we seek partitions where each part has at most one point of each color. In this manuscript, we bound the number of color classes needed for the existence of partitions where the convex hulls of the parts intersect even after any set of $t$ colors is removed. We prove asymptotically optimal bounds for $t$ when $r \le d+1$, improve known bounds when $r>d+1$, and give a geometric characterization for the configurations of points for which $t=N-o(N)$.