The concept of formal duality was proposed by Cohn, Kumar and Schürmann, which reflects a remarkable symmetry among energy-minimizing periodic configurations. This formal duality was later translated into a purely combinatorial property by Cohn, Kumar, Reiher and Schürmann, where the corresponding combinatorial objects were called formally dual pairs. So far, except the results presented in Li and Pott (arXiv:1810.05433v3), we have little information about primitive formally dual pairs having subsets with unequal sizes. In this paper, we propose a direct construction of primitive formally dual pairs having subsets with unequal sizes in $\mathbb{Z}_2 \times \mathbb{Z}_4^{2m}$, where $m \ge 1$. This construction recovers an infinite family obtained in Li and Pott (arXiv:1810.05433v3), which was derived by employing a recursive approach. Although the resulting infinite family was known before, the idea of the direct construction is new and provides more insights which were not known from the recursive approach.