A countable discrete group $G$ is called Choquet-Deny if for every non-degenerate probability measure $μ$ on $G$ it holds that all bounded $μ$-harmonic functions are constant. We show that a finitely generated group $G$ is Choquet-Deny if and only if it is virtually nilpotent. For general countable discrete groups, we show that $G$ is Choquet-Deny if and only if none of its quotients has the infinite conjugacy class property. Moreover, when $G$ is not Choquet-Deny, then this is witnessed by a symmetric, finite entropy, non-degenerate measure.