Agarwal introduced $n$-color compositions in 2000 and most subsequent research has focused on restricting which parts are allowed. Here we focus instead on restricting allowed colors. After three general results, giving recurrence formulas for the cases of given allowed colors, given prohibited colors, and colors satisfying modular conditions, we consider several more specific conditions, establishing direct formulas and connections to other combinatorial objects. Proofs are combinatorial, mostly using the notion of spotted tilings introduced by the first named author in 2012.