It is known that the sectional genus of a polarized variety has an upper bound, which is an extension of the Castelnuovo bound on the genus of a projective curve. Polarized varieties whose sectional genus achieves this bound are called Castelnuovo. On the other hand, a lattice polytope is called Castelnuovo if the associated polarized toric variety is Castelnuovo. Kawaguchi characterized Castelnuovo polytopes having interior lattice points in terms of their $h^*$-vectors. In this paper, as a generalization of this result, a characterization of all Castelnuovo polytopes will be presented. Finally, as an application of our characterization, we give a sufficient criterion for a lattice polytope to be IDP.