We study the behavior of $h$-vectors associated to matroid complexes under weak maps, or inclusions of matroid polytopes. Specifically, we show that the $h$-vector of the order complex of the lattice of flats of a matroid is component-wise non-increasing under a weak map. This result extends to the flag $h$-vector. We note that the analogous result also holds for independence complexes and rank-preserving weak maps.