Let $p$ denote a prime number. In this note, we focus on the modular Terwilliger algebras of association schemes defined in [3]. We define the primary module of a modular Terwilliger algebra of an association scheme and determine all its composition factors up to isomorphism. We then characterize the $p'$-valenced association schemes by some properties of their modular Terwilliger algebras. The corollaries about the modular Terwilliger algebras of association schemes are given.