Hybrid precoding, which consists of an RF precoder and a baseband precoder, is a popular precoding architecture for massive MIMO due to its low hardware cost and power consumption. In conventional hybrid precoding, both RF and baseband precoders are adaptive to the real-time channel state information (CSI). As a result, an individual RF precoder is required for each subcarrier in wideband systems, leading to high implementation cost. To overcome this issue, two-timescale hybrid precoding (THP), which adapts the RF precoder to the channel statistics, has been proposed. Since the channel statistics are approximately the same over different subcarriers, only a single RF precoder is required in THP. Despite the advantages of THP, there lacks a unified and efficient algorithm for its optimization due to the non-convex and stochastic nature of the problem. Based on stochastic successive convex approximation (SSCA), we propose an online algorithmic framework called SSCA-THP for general THP optimization problems, in which the hybrid precoder is updated by solving a quadratic surrogate optimization problem whenever a new channel sample is obtained. Then we prove the convergence of SSCA-THP to stationary points. Finally, we apply SSCA-THP to solve three important THP optimization problems and verify its advantages over existing solutions.