We study the number of $s$-element subsets $J$ of a given abelian group $G$, such that $|J+J|\leq K|J|$. Proving a conjecture of Alon, Balogh, Morris and Samotij, and improving a result of Green and Morris, who proved the conjecture for $K$ fixed, we provide an upper bound on the number of such sets which is tight up to a factor of $2^{o(s)},$ when $G=\mathbb{Z}$ and $K=o(s/(\log n)^3)$. We also provide a generalization of this result to arbitrary abelian groups which is tight up to a factor of $2^{o(s)}$ in many cases. The main tool used in the proof is the asymmetric container lemma, introduced recently by Morris, Samotij and Saxton.