If $H$ is a numerical semigroup (that is, a cofinite subset of the non-negative integers closed under addition), then the non-empty subsets of $H$ form a semigroup $\mathcal P(H)$ under the sumset operation induced by addition in $H$. Moreover, if $0 \in H$, then $\mathcal P(H)$ is a monoid with identity element $\{0\}$, and the family $\mathcal P_0(H)$ of all subsets of $H$ containing $0$ is a submonoid of $\mathcal P(H)$. We show that the automorphism group of $\mathcal P(H)$ is trivial, and the same holds for $\mathcal P_0(H)$ when $0 \in H$. The proofs blend ideas from combinatorics and semigroup theory.