Given a surface with boundary and some points on its boundary, a polygon diagram is a way to connect those points as vertices of non-overlapping polygons on the surface. Such polygon diagrams represent non-crossing permutations on a surface with any genus and number of boundary components. If only bigons are allowed, then it becomes an arc diagram. The count of arc diagrams is known to have a rich structure. We show that the count of polygon diagrams exhibits the same interesting behaviours, in particular it is almost polynomial in the number of points on the boundary components, and the leading coefficients of those polynomials are the intersection numbers on the compactified moduli space of curves.