Let $M$ be a complete connected Riemannian manifold with boundary $\partial M$, and let $P_t$ be the Neumann semigroup generated by $\frac{ 1}{ 2} L$ where $L=Δ+Z$ for a $C^1$-vector field $Z$ on $M$. We establish Bismut type formulae for $LP_t f$ and ${\rm Hess}_{P_tf}$ and present estimates of these quantities under suitable curvature conditions. In case when $P_t$ is symmetric in $L^2(μ)$ for some probability measure $μ$, a new type of log-Sobolev inequality is established which links the relative entropy $H$, the Stein discrepancy $S$, and relative Fisher information $I$, generalizing the authors' recent work in the case without boundary.