The concept of formal duality was proposed by Cohn, Kumar and Schürmann, which reflects a remarkable symmetry among energy-minimizing periodic configurations. This formal duality was later on translated into a purely combinatorial property by Cohn, Kumar, Reiher and Schürmann, where the corresponding combinatorial objects were called formally dual pairs. Almost all known examples of primitive formally dual pairs satisfy that the two subsets have the same size. Indeed, prior to this work, there was only one known example having subsets with unequal sizes in $\mathbb{Z}_2 \times \mathbb{Z}_4^2$. Motivated by this example, we propose a lifting construction framework and a recursive construction framework, which generate new primitive formally dual pairs from known ones. As an application, for $m \ge 2$, we obtain $m+1$ pairwise inequivalent primitive formally dual pairs in $\mathbb{Z}_2 \times \mathbb{Z}_4^{2m}$, which have subsets with unequal sizes.