We make the observation that certain group automorphisms that fix a large subgroup of an abelian group cannot be multipliers in any non-trivial abelian difference sets, with the single exception of an involution that can be a multiplier in Hadamard difference sets, provided that the difference set contains a sub-difference set of the same type. We use this observation together with a multiplier theorem to rule out the existence of difference sets, and derive bounds for the numerical multiplier group of a difference set.