Recently, terahertz (THz) communication has drawn considerable attention as one of the promising technologies for the future wireless communications owning to its ultra-wide bandwidth. Nonetheless, one major obstacle that prevents the actual deployment of THz lies in its inherent huge attenuation. Intelligent reflecting surface (IRS) and multiple-input multiple-output (MIMO) represent two effective solutions for compensating the large pathloss in THz systems. In this paper, we consider an IRS-aided multi-user THz MIMO system with orthogonal frequency division multiple access, where the sparse radio frequency chain antenna structure is adopted for reducing the power consumption. The objective is to maximize the weighted sum rate via jointly optimizing the hybrid analog/digital beamforming at the base station and reflection matrix at the IRS. {Since the analog beamforming and reflection matrix need to cater all users and subcarriers, it is difficult to directly solve the formulated problem, and thus, an alternatively iterative optimization algorithm is proposed.} Specifically, the analog beamforming is designed by solving a MIMO capacity maximization problem, while the digital beamforming and reflection matrix optimization are both tackled using semidefinite relaxation technique. Considering that obtaining perfect channel state information (CSI) is a challenging task in IRS-based systems, we further explore the case with the imperfect CSI for the channels from the IRS to users. Under this setup, we propose a robust beamforming and reflection matrix design scheme for the originally formulated non-convex optimization problem. Finally, simulation results are presented to demonstrate the effectiveness of the proposed algorithms.