Following earlier work by Coons-Maraj-Misra-Sorea and Misra-Sullivant, we study colored, undirected Gaussian graphical models, and present a necessary and sufficient condition for such a model to have binomial vanishing ideal. These conditions involve Jordan schemes, a variant of association schemes, well-known structures in algebraic combinatorics. Using association schemes without transitive group action, we refute the conjecture by Coons-Maraj-Misra-Sorea that binomiality implies that the color classes must be orbits under the automorphism group of the colored graph.