The symmetry of polygons can be characterized by the number of symmetry axes they have. For $n$-polygons with $p$ or $p^2$ vertices $p\geq3$ there exist few symmetry categories, depending from the number of symmetry-axes the have. Further in the case of $n=p^2$ there exist $p^2$-polygons with no axis, but the property of $p$-circularity. This work investigates the corresponding equivalence classes and their enumeration. The total number of equivalence classes and the number of the regular ones are known. Formulas get established for the number of equivalence classes for the other categories of symmetry. We show complete sets of representatifs in some cases. Together, these results provide a comprehensive description of the symmetry structure of polygons with a prime number or prime square number of vertices and lay the groundwork for extending the classification and enumeration to polygons with a more composed numbers of vertices.