Non-orthogonal multiple access based cooperative relaying system (NOMA-CRS) has been proposed to alleviate the decay in spectral efficiency of the conventional CRS. However, existing NOMA-CRS studies assume perfect successive interference canceler at the relay and mostly investigate sum-rate whereas the error performance has not been taken into consideration. In this paper, we analyze error performance of the NOMA-CRS and the closed-form bit error probability (BEP) expression is derived over Nakagami-m fading channels. Then, thanks to the high performance of machine learning (ML) in challenging optimization problems, a joint power sharing-power allocation (PS-PA) scheme is proposed to minimize the bit error rate (BER) of the NOMA-CRS. The proposed ML-assisted optimization has a very low online implementation complexity. Based on provided extensive simulations, theoretical BEP analysis is validated. Besides, the proposed ML-aided PS-PA provides minimum BER (MBER) and outperforms previous PA strategies for the NOMA-CRS notably.