Hybrid analog and digital beamforming is a promising candidate for large-scale mmWave MIMO systems because of its ability to significantly reduce the hardware complexity of the conventional fully-digital beamforming schemes while being capable of approaching the performance of fully-digital schemes. Most of the prior work on hybrid beamforming considers narrowband channels. However, broadband systems such as mmWave systems are frequency-selective. In broadband systems, it is desirable to design common analog beamformer for the entire band while employing different digital beamformers in different frequency sub-bands. This paper considers hybrid beamforming design for systems with OFDM modulation. First, for a SU-MIMO system where the hybrid beamforming architecture is employed at both transmitter and receiver, we show that hybrid beamforming with a small number of RF chains can asymptotically approach the performance of fully-digital beamforming for a sufficiently large number of transceiver antennas due to the sparse nature of the mmWave channels. For systems with a practical number of antennas, we then propose a unified heuristic design for two different hybrid beamforming structures, the fully-connected and the partially-connected structures, to maximize the overall spectral efficiency of a mmWave MIMO system. Numerical results are provided to show that the proposed algorithm outperforms the existing hybrid beamforming methods and for the fully-connected architecture the proposed algorithm can achieve spectral efficiency very close to that of the optimal fully-digital beamforming but with much fewer RF chains. Second, for the MU-MISO case, we propose a heuristic hybrid percoding design to maximize the weighted sum rate in the downlink and show numerically that the proposed algorithm with practical number of RF chains can already approach the performance of fully-digital beamforming.