We initiate the study of spanners in arbitrarily vertex- or edge-colored graphs (with no "legality" restrictions), that are resilient to failures of entire color classes. When a color fails, all vertices/edges of that color crash. An $f$-color fault-tolerant ($f$-CFT) $t$-spanner of an $n$-vertex colored graph $G$ is a subgraph $H$ that preserves distances up to factor $t$, even in the presence of at most $f$ color faults. This notion generalizes the well-studied $f$-vertex/edge fault-tolerant ($f$-V/EFT) spanners. The size of an $f$-V/EFT spanner crucially depends on the number $f$ of vertex/edge faults to be tolerated. In the colored variants, even a single color fault can correspond to an unbounded number of vertex/edge faults. The key conceptual contribution of this work is in showing that the size (number of edges) required by an $f$-CFT spanner is in fact comparable to its uncolored counterpart, with no dependency on the size of color classes. We provide optimal bounds on the size required by $f$-CFT spanners, revealing an interesting phenomenon: while (individual) edge faults are "easier" than vertex faults in terms of spanner size, edge-color faults are "harder" than vertex-color faults. Our upper bounds are based on a generalization of the blocking set technique of [Bodwin and Patel, PODC 2019] for analyzing the (exponential-time) greedy algorithm for FT spanners. We complement them by providing efficient constructions of CFT spanners with similar size guarantees, based on the algorithm of [Dinitz and Robelle, PODC 2020].