Inspired by an experimental study of energy-minimizing periodic configurations in Euclidean space, Cohn, Kumar and Schürmann proposed the concept of formal duality between a pair of periodic configurations, which indicates an unexpected symmetry possessed by the energy-minimizing periodic configurations. Later on, Cohn, Kumar, Reiher and Schürmann translated the formal duality between a pair of periodic configurations into the formal duality of a pair of subsets in a finite abelian group. This insight suggests to study the combinatorial counterpart of formal duality, which is a configuration named formally dual pair. In this paper, we initiate a systematic investigation on formally dual pairs in finite abelian groups, which involves basic concepts, constructions, characterizations and nonexistence results. In contrast to the belief that primitive formally dual pairs are very rare in cyclic groups, we construct three families of primitive formally dual pairs in noncyclic groups. These constructions enlighten us to propose the concept of even sets, which reveals more structural information about formally dual pairs and leads to a characterization of rank three primitive formally dual pairs. Finally, we derive some nonexistence results about primitive formally dual pairs, which are in favor of the main conjecture that except two small examples, no primitive formally dual pair exists in cyclic groups.