Massive developments in mobile wireless telecommunication networks have been made during the last few decades. At present, mobile users are getting familiar with the latest 5G networks, and the discussion for the next generation of Beyond 5G (B5G)/6G networks has already been initiated. It is expected that B5G/6G will push the existing network capabilities to the next level, with higher speeds, enhanced reliability and seamless connectivity. To make these expectations a reality, research is progressing on new technologies, architectures, and intelligence-based decision-making processes related to B5G/6G. Privacy considerations are a crucial aspect that requires further attention in such developments, as billions of people and devices will be transmitting data through the upcoming network. However, the main recognition remains biased towards the network security. A discussion focused on privacy of B5G/6G is lacking at the moment. To address the gap, this paper provides a comprehensive survey on privacy-related aspects of B5G/6G networks. First, it discusses a taxonomy of different privacy perspectives. Based on the taxonomy, the paper then conceptualizes a set of challenges that appear as barriers to reach privacy preservation. Next, this work provides a set of solutions applicable to the proposed architecture of B5G/6G networks to mitigate the challenges. It also provides an overview of standardization initiatives for privacy preservation. Finally, the paper concludes with a roadmap of future directions, which will be an arena for new research towards privacy-enhanced B5G/6G networks. This work provides a basis for privacy aspects that will significantly impact peoples' daily lives when using these future networks.