In this study we derive novel optimal algorithms for joint power control and beamforming design in modern large-scale MIMO systems, such as those based on the cell-free massive MIMO and XL-MIMO concepts. In particular, motivated by the need for scalable system architectures, we formulate and solve nontrivial two-timescale extensions of the classical uplink power minimization and max-min fair resource allocation problems. In our formulations, we let the beamformers be functions mapping partial instantaneous channel state information (CSI) to beamforming weights, and we jointly optimize these functions and the power control coefficients based on long-term statistical CSI. This long-term approach mitigates the severe scalability issues of competing short-term iterative algorithms in the literature, where a central controller endowed with global instantaneous CSI must solve a complex optimization problem for every channel realization, hence imposing very demanding requirements in terms of computational complexity and signaling overhead. Moreover, our approach outperforms the available long-term approaches, which do not jointly optimize powers and beamformers. The obtained optimal long-term algorithms are then illustrated and compared against existing short-term and long-term algorithms via numerical simulations in a cell-free massive MIMO setup with different levels of cooperation.