In this paper, a joint transmitter and receiver design for pattern division multiple access (PDMA) is proposed. At the transmitter, pattern mapping utilizes power allocation to improve the overall sum rate, and beam allocation to enhance the access connectivity. At the receiver, hybrid detection utilizes a spatial filter to suppress the inter-beam interference caused by beam domain multiplexing, and successive interference cancellation to remove the intra-beam interference caused by power domain multiplexing. Furthermore, we propose a PDMA joint design approach to optimize pattern mapping based on both the power domain and beam domain. The optimization of power allocation is achieved by maximizing the overall sum rate, and the corresponding optimization problem is shown to be convex theoretically. The optimization of beam allocation is achieved by minimizing the maximum of the inner product of any two beam allocation vectors, and an effective dimension reduction method is proposed through the analysis of pattern structure and proper mathematical manipulations. Simulation results show that the proposed PDMA approach outperforms the orthogonal multiple access and power-domain non-orthogonal multiple access approaches even without any optimization of pattern mapping, and that the optimization of beam allocation yields a significant performance improvement than the optimization of power allocation.