Several recent papers have examined a rational polyhedron $P_m$ whose integer points are in bijection with the numerical semigroups (cofinite, additively closed subsets of the non-negative integers) containing $m$. A combinatorial description of the faces of $P_m$ was recently introduced, one that can be obtained from the divisibility posets of the numerical semigroups a given face contains. In this paper, we study the faces of $P_m$ containing arithmetical numerical semigroups and those containing certain glued numerical semigroups, as an initial step towards better understanding the full face structure of $P_m$. In most cases, such faces only contain semigroups from these families, yielding a tight connection to the geometry of $P_m$.