We show that the set of entries generated by any finite set of doubly stochastic matrices is nowhere dense, in contrast to the cases of stochastic matrices or unitary matrices. In other words, there is no finite universal set of doubly stochastic matrices, even with the weakest notion of universality. Our proof is based on a theorem for topological semigroups with the convergent property. A topological semigroup is convergent if every infinite product converges. We show that in a compact and convergent semigroup, under some restrictions, the closure of every finitely generated subsemigroup can be instead generated directly by the generating set and the limits of infinite products.