We develop a model for the stability and maintenance of phonological categories. Examples of phonological categories are vowel sounds such as "i" and "e". We model such categories as consisting of collections of labeled exemplars that language users store in their memory. Each exemplar is a detailed memory of an instance of the linguistic entity in question. Starting from an exemplar-level model we derive integro-differential equations for the long-term evolution of the density of exemplars in different portions of phonetic space. Using these latter equations we investigate under what conditions two phonological categories merge or not. Our main conclusion is that for the preservation of distinct phonological categories, it is necessary that anomalous speech tokens of a given category are discarded, and not merely stored in memory as an exemplar of another category.