In view of noteworthy communications performance improvements for future B5G/6G (such as cognitive Internet of Things, space-ground integration network and so on), cooperative communications (CC) diversity with relays selection algorithms have been extensively studied to significantly improve communications quality, but so far there is still a lot of potential optimization work with CC schemes. In this paper, in the light of NP-hard problem of subsets relays selection, further studies for theorems of relays subsets with K-layers power allocation standard have been put forward to explore better performance in B5G/6G cognitive IoT (Internet of Things) networks, we propose unified layers-based optimized mobile relays subsets algorithms for full-duplex (FD) non-orthogonal multiple access (NOMA) to greatly improve transmission rate. After revealing and taking into account fundamental properties of relays, such as mobile relays nodes state, relays locations, fading characteristics and so on, optimized FD-NOMA algorithm based on these relays features has been presented to improve transmission validity, and a related series of relays subsets theorems have been derived and proved, then minimum upper bound of maximum transmission rates has been estimated to reveal two-way balanced optimal transmission conclusion for FD-NOMA. In general, proposed general and optimized algorithm can be used in multiple future cooperative communications scenarios in B5G/6G networks such as cognitive IoT. Simulations results show that proposed scheme has several times transmission rates than other classical relays selection algorithms