We prove the following sharp upper bound for the gradient of the Neumann semigroup $P_t$ on a $d$-dimensional compact domain $\OO$ with boundary either $C^2$-smooth or convex: $$\|\nn P_t\|_{1\to \infty}\le \ff{c}{t^{(d+1)/2}},\ \ t>0,$$ where $c>0$ is a constant depending on the domain and $\|\cdot\|_{1\to\infty}$ is the operator norm from $L^1(\OO)$ to $L^\infty(\OO)$. This estimate implies a Gaussian type point-wise upper bound for the gradient of the Neumann heat kernel, which is applied to the study of the Hardy spaces, Riesz transforms, and regularity of solutions to the inhomogeneous Neumann problem on compact convex domains.