Suppose $c_n(σ)$ denotes the number of cyclic permutations in $\mathcal{S}_n$ that avoid a pattern $σ$. In this paper, we define partial groupoid structures on cyclic pattern-avoiding permutations that allow us to build larger cyclic pattern-avoiding permutations from smaller ones. We use this structure to find recursive lower bounds on $c_n(σ)$. These bounds imply that $c_n(σ)$ has a growth rate of at least 3 for $σ\in\{231,312,321\}$ and a growth rate of at least 2.6 for $σ\in\{123,132,213\}$. In the process, we prove (and sometimes improve) a conjecture of Bóna and Cory that $c_n(σ)\geq 2 c_{n-1}(σ)$ for all $σ\in\mathcal{S}_3\setminus\{123\}$ and $n\geq 2.$