Handling multiplicity without losing much power has been a persistent challenge in various fields that often face the necessity of managing numerous statistical tests simultaneously. Recently, $p$-value combination methods based on heavy-tailed distributions, such as a Cauchy distribution, have received much attention for their ability to handle multiplicity without the prescribed knowledge of the dependence structure. This paper delves into these types of $p$-value combinations through the lens of extreme value theory. Distributions with regularly varying tails, a subclass of heavy tail distributions, are found to be useful in constructing such $p$-value combinations. Three $p$-value combination statistics (sum, max cumulative sum, and max) are introduced, of which left tail probabilities are shown to be approximately uniform when the global null is true. The primary objective of this paper is to bridge the gap between current developments in $p$-value combination methods and the literature on extreme value theory, while also offering guidance on selecting the calibrator and its associated parameters.