We study Cohen-Macaulay Hopf monoids in the category of species. The goal is to apply techniques from topological combinatorics to the study of polynomial invariants arising from combinatorial Hopf algebras. Given a polynomial invariant arising from a linearized Hopf monoid, we show that under certain conditions it is the Hilbert polynomial of a relative simplicial complex. If the Hopf monoid is Cohen-Macaulay, we give necessary and sufficient conditions for the corresponding relative simplicial complex to be relatively Cohen-Macaulay, which implies that the polynomial has a nonnegative $h$-vector. We apply our results to the weak and strong chromatic polynomials of acyclic mixed graphs, and the order polynomial of a double poset.