The notion of $S$-labeling of graphs, where $S$ is a subset of a symmetric group, was introduced in 2019 by Jin, Wong, and Zhu. This notion provides the framework for a common generalization of various well studied notions of graph coloring, including classical coloring, signed $k$-coloring, signed $\mathbb{Z}_k$-coloring, DP (or correspondence) coloring, group coloring, and coloring of gained graphs. In this paper, we present a unified and simple polynomial method for giving exponential lower bounds on the number of colorings of an $S$-labeled graph for all such $S$. This algebraic technique allows us to prove new lower bounds on the number of colorings of any $S$-labeling of graphs satisfying certain sparsity conditions. We also investigate how the structure of $S$ can be exploited to improve the applicability of these bounds. Our results give new lower bounds on the number of DP-colorings, and consequently the number of all types of colorings listed above. This includes the chromatic polynomial and the number of list colorings of families of planar graphs, and the number of colorings of signed graphs. These enumerative bounds improve previously known results or are the first such known results.