While two-way relaying is a promising way to enhance the spectral efficiency of wireless networks, the imbalance of relay-user distances may lead to excessive wireless power at the nearby-users. To exploit the excessive power, the recently proposed harvest-then-transmit technique can be applied. However, it is well-known that harvest-then-transmit introduces uplink-downlink coupling for a user. Together with the co-dependent relationship between paired users and interference among multiple user pairs, wirelessly powered two-way relay network suffers from the unique pairwise uplink-downlink coupling, and the joint uplink-downlink network design is nontrivial. To this end, for the one pair users case, we show that a global optimal solution can be obtained. For the general case of multi-pair users, based on the rank-constrained difference of convex program, a convergence guaranteed iterative algorithm with an efficient initialization is proposed. Furthermore, a lower bound to the performance of the optimal solution is derived by introducing virtual receivers at relay. Numerical results on total transmit power show that the proposed algorithm achieves a transmit power value close to the lower bound.