The performance of data delivery in wireless relay networks (WRNs), such as delay-tolerant networks and device-to-device communications heavily relies on the cooperation of mobile nodes (i.e., users and their carried devices). However, selfish nodes may refuse to relay data to others or share their resources with them due to various reasons, such as resource limitations or social preferences. Meanwhile, misbehaving nodes can launch different types of internal attacks (e.g., blackhole and trust-related attacks) to disrupt the normal operation of the network. Numerous mechanisms have been recently proposed to establish secure and efficient communications in WRNs in the presence of selfish and malicious nodes (referred as non-cooperative WRNs). In this paper, we present an in-depth survey on human-centric communication challenges and solutions in the non-cooperative WRNs that focuses on: (1) an overview of the non-cooperative WRNs and introduction to various types of node selfish and malicious behaviors, (2) the impact analysis of node selfish and malicious behaviors on the performance of data forwarding and distribution, (3) selfish and malicious node detection and defense systems, and (4) incentive mechanisms. Finally, we discuss several open problems and future research challenges.