In recent years, the multiple-input multiple-output (MIMO) non-orthogonal multiple-access (NOMA) systems have attracted a significant interest in the relevant research communities. As a potential precoding scheme, the generalized singular value decomposition (GSVD) can be adopted in MIMO-NOMA systems and has been proved to have high spectral efficiency. In this paper, the performance of the GSVD-based MIMO-NOMA communications with Rician fading is studied. In particular, the distribution characteristics of generalized singular values (GSVs) of channel matrices are analyzed. Two novel mathematical tools, the linearization trick and the deterministic equivalent method, which are based on operator-valued free probability theory, are exploited to derive the Cauchy transform of GSVs. An iterative process is proposed to obtain the numerical values of the Cauchy transform of GSVs, which can be exploited to derive the average data rates of the communication system. In addition, the special case when the channel is modeled as Rayleigh fading, i.e., the line-of-sight propagation is trivial, is analyzed. In this case, the closed-form expressions of average rates are derived from the proposed iterative process. Simulation results are provided to validate the derived analytical results.