Given an infinite word over the alphabet $\{0,1,2,3\}$, we define a class of bipartite hereditary graphs $\mathcal{G}^α$, and show that $\mathcal{G}^α$ has unbounded clique-width unless $α$ contains at most finitely many non-zero letters. We also show that $\mathcal{G}^α$ is minimal of unbounded clique-width if and only if $α$ belongs to a precisely defined collection of words $Γ$. The set $Γ$ includes all almost periodic words containing at least one non-zero letter, which both enables us to exhibit uncountably many pairwise distinct minimal classes of unbounded clique width, and also proves one direction of a conjecture due to Collins, Foniok, Korpelainen, Lozin and Zamaraev. Finally, we show that the other direction of the conjecture is false, since $Γ$ also contains words that are \emph{not} almost periodic.