It is a classical result of Kaimanovich and Vershik and independently of Rosenblatt that a non-amenable group admits a non-degenerate symmetric measure such that the Poisson boundary is trivial. Most if not all examples to date of non-free actions of countable groups on their Poisson boundaries had the stabilizers sitting inside the amenable radical. We show that every countable non-C*-simple group admits a symmetric measure of full support with non-trivial stabilizers. For a class of non-C*-simple groups with trivial amenable radical, which is non-empty as was shown by le Boudec, this gives a wealth of examples with non-normal stabilizers.