The cloud radio access network (C-RAN) concept, in which densely deployed access points (APs) are empowered by cloud computing to cooperatively support mobile users (MUs), to improve mobile data rates, has been recently proposed. However, the high density of active ("on") APs results in severe interference and also inefficient energy consumption. Moreover, the growing popularity of highly interactive applications with stringent uplink (UL) requirements, e.g. network gaming and real-time broadcasting by wireless users, means that the UL transmission is becoming more crucial and requires special attention. Therefore in this paper, we propose a joint downlink (DL) and UL MU-AP association and beamforming design to coordinate interference in the C-RAN for energy minimization, a problem which is shown to be NP hard. Due to the new consideration of UL transmission, it is shown that the two state-of-the-art approaches for finding computationally efficient solutions of joint MU-AP association and beamforming considering only the DL, i.e., group-sparse optimization and relaxed-integer programming, cannot be modified in a straightforward way to solve our problem. Leveraging on the celebrated UL-DL duality result, we show that by establishing a virtual DL transmission for the original UL transmission, the joint DL and UL optimization problem can be converted to an equivalent DL problem in C-RAN with two inter-related subproblems for the original and virtual DL transmissions, respectively. Based on this transformation, two efficient algorithms for joint DL and UL MU-AP association and beamforming design are proposed, whose performances are evaluated and compared with other benchmarking schemes through extensive simulations.