We show that the $f$-vector of Galashin's poset associahedron $\mathscr A(P)$ only depends on the comparability graph of $P$. In particular, this allows us to produce a family of polytopes with the same $f$-vectors as permutohedra, but that are not combinatorially equivalent to permutohedra.