Non-terrestrial network (NTN) communication has garnered considerable attention from government entities, industries, and academia in recent times. NTN networks encompass a variety of systems, including Low Earth Orbit (LEO) satellites, Medium Earth Orbit (MEO) satellites, Geostationary Earth Orbit (GEO) satellites, High Altitude Platforms (HAPS), and Low Altitude Platforms (LAPS). Furthermore, the deployment of high-throughput satellites (HTS/VHTS) in the GEO space has gained momentum. While LEO and MEO satellites offer advantages such as low latency and reduced launching costs compared to GEO satellites, this study focuses on GEO satellites due to their stationary nature and broader coverage. In traditional cellular networks, each user equipment (UE) is allocated at least one resource block (RB), which is not shared with other UEs. However, in NTN communications, where the coverage area is extensive, dedicating an RB to only one UE is an inefficient utilization of radio resources. To address this challenge, fractional programming (FP), cognitive radio, and rate splitting multiple access (RSMA) are existing technologies. This paper aims to maximize spectral efficiency, average RBG rate, and sum rate for GEO satellite systems. However, achieving this objective involves dealing with a non-convex, NP-hard problem, as it requires the logarithmic sum of different fractions. Finding a global solution to such an NP-hard problem presents significant challenges. This paper introduces a novel alternate fractional programming algorithm specifically designed to tackle these complex NP-hard problems in the context of GEO NTN cellular networks. By employing this innovative approach, the study seeks to contribute to the optimization of NTN communication systems, enabling efficient resource allocation and improved network performance.