An $r$-dual tower of groups is a nested sequence of finite groups, like the symmetric groups, whose Bratteli diagram forms an $r$-dual graded graph. Miller and Reiner introduced a special case of these towers in order to study the Smith forms of the up and down maps in a differential poset. Agarwal and the author have also used these towers to compute critical groups of representations of groups appearing in the tower. In this paper I prove that when $r$ is one or prime, wreath products of a fixed group with the symmetric groups are the only $r$-dual tower of groups, and conjecture that this is the case for general values of $r$. This implies that these wreath products are the only groups for which one can define an analog of the Robinson-Schensted bijection in terms of a growth rule in a dual graded graph.