We show that the number of numerical semigroups with multiplicity three, four or five and fixed genus is increasing as a function in the genus. To this end we use the Kunz polytope for these multiplicities. Counting numerical semigroups with fixed multiplicity and genus is then an integer partition problem with some extra conditions (those of membership to the Kunz polytope). For the particular case of multiplicity four, we are able to prove that the number of numerical semigroups with multiplicity four and genus $g$ is the number of partitions $x+y+z=g+6$ with $0<x\le y\le z$, $x\neq 1$, $y\neq 2$ and $z\neq 3$.