A Cayley permutation is a word of positive integers such that if a letter appears in this word, then all positive integers smaller than that letter also appear. We initiate a systematic study of pattern avoidance on Cayley permutations adopting a combinatorial species approach. Our methods lead to species equations, generating series, and counting formulas for Cayley permutations avoiding any pattern of length at most three. We also introduce the species of primitive structures as a generalization of Cayley permutations with no "flat steps". Finally, we explore various notions of Wilf equivalence arising in this context.