This paper considers the joint design of user power allocation and relay beamforming in relaying communications, in which multiple pairs of single-antenna users exchange information with each other via multiple-antenna relays in two time slots. All users transmit their signals to the relays in the first time slot while the relays broadcast the beamformed signals to all users in the second time slot. The aim is to maximize the system's energy efficiency (EE) subject to quality-of-service (QoS) constraints in terms of exchange throughput requirements. The QoS constraints are nonconvex with many nonlinear cross-terms, so finding a feasible point is already computationally challenging. The sum throughput appears in the numerator while the total consumption power appears in the denominator of the EE objective function. The former is a nonconcave function and the latter is a nonconvex function, making fractional programming useless for EE optimization. Nevertheless, efficient iterations of low complexity to obtain its optimized solutions are developed. The performances of the multiple-user and multiple-relay networks under various scenarios are evaluated to show the merit of the paper development.