Terahertz (THz) communications are regarded as a pillar technology for the sixth generation (6G) wireless systems, by offering multi-ten-GHz bandwidth. To overcome the short transmission distance and huge propagation loss, ultra-massive (UM) MIMO systems that employ sub-millimeter wavelength antennas array are proposed to enable an enticingly high array gain. In the UM-MIMO systems, hybrid beamforming stands out for its great potential in promisingly high data rate and reduced power consumption. In this paper, challenges and features of the THz hybrid beamforming design are investigated, in light of the distinctive THz peculiarities. Specifically, we demonstrate that the spatial degree-of-freedom (SDoF) is less than 5, which is caused by the extreme sparsity of the THz channel. The blockage problem caused by the huge reflection and scattering losses, as high as 15 dB or over, is studied. Moreover, we analyze the challenges led by the array containing 1024 or more antennas, including the requirement for intelligent subarray architecture, strict energy efficiency, and propagation characterization based on spherical-wave propagation mechanisms. Owning up to hundreds of GHz bandwidth, beam squint effect could cause over 5~dB array gain loss, when the fractional bandwidth exceeds 10%. Inspired by these facts, three novel THz-specific hybrid beamforming architectures are presented, including widely-spaced multi-subarray, dynamic array-of-subarrays, and true-time-delay-based architectures. We also demonstrate the potential data rate, power consumption, and array gain capabilities for THz communications. As a roadmap of THz hybrid beamforming design, multiple open problems and potential research directions are elaborated.