We establish relations between different characterizations of order in spin glass models. We first prove that the broadening of the replica overlap distribution indicated by a nonzero standard deviation of the replica overlap $R^{1,2}$ implies the non-differentiability of the two-replica free energy with respect to the replica coupling parameter $λ$. In $\mathbb Z_2$ invariant models such as the standard Edwards-Anderson model, the non-differentiability is equivalent to the spin glass order characterized by a nonzero Edwards-Anderson order parameter. This generalization of Griffiths' theorem is proved for any short-range spin glass models with classical bounded spins. We also prove that the non-differentiability of the two-replica free energy mentioned above implies replica symmetry breaking in the literal sense, i.e., a spontaneous breakdown of the permutation symmetry in the model with three replicas. This is a general result that applies to a large class of random spin models, including long-range models such as the Sherrington-Kirkpatrick model and the random energy model.