Non-orthogonal multiple access (NOMA) is a promising multiple access technique. Its performance depends strongly on the wireless channel property, which can be enhanced by reconfigurable intelligent surfaces (RISs). In this paper, we jointly optimize base station (BS) precoding and RIS configuration with unsupervised machine learning (ML), which looks for the optimal solution autonomously. In particular, we propose a dedicated neural network (NN) architecture RISnet inspired by domain knowledge in communication. Compared to state-of-the-art, the proposed approach combines analytical optimal BS precoding and ML-enabled RIS, has a high scalability to control more than 1000 RIS elements, has a low requirement for channel state information (CSI) in input, and addresses the mutual coupling between RIS elements. Beyond the considered problem, this work is an early contribution to domain knowledge enabled ML, which exploit the domain expertise of communication systems to design better approaches than general ML methods.