Intelligent reflecting surfaces (IRSs) are regarded as promising enablers for future millimeter wave (mmWave) wireless communication, due to their ability to create favorable line-of-sight (LoS) propagation environments. In this paper, we investigate channel estimation in downlink IRS-assisted mmWave multiple-input multiple-output (MIMO) systems. By leveraging the sparsity of mmWave channels, we formulate the channel estimation problem as a fixed-rank constrained non-convex optimization problem. To tackle the non-convexity, an efficient algorithm is proposed by capitalizing on alternating minimization and manifold optimization (MO), which yields a locally optimal solution. Simulation results show that the proposed MO-based estimation (MO-EST) algorithm significantly outperforms two benchmark schemes and demonstrate the robustness of the MO-EST algorithm with respect to imperfect knowledge of the sparsity level of the channels in practical implementations.