We investigate the joint uplink-downlink configuration of an intelligent reflecting surface (IRS) for multi-user frequency-division-duplexing (FDD) and time-division-duplexing (TDD) systems. This is motivated in FDD since uplink and downlink transmissions occur simultaneously and hence an IRS must be jointly configured for both transmissions. In TDD, while a joint design is not strictly necessary, it can significantly reduce feedback overhead, power consumption, and configuration periods associated with updating the IRS. To compute the trade-off between uplink and downlink rates achieved by a joint design, a weighted-sum problem is formulated and optimized using a developed block-coordinate descent algorithm. The resulting uplink-downlink trade-off regions are investigated by numerical simulation to gain insights into different scenarios. In all FDD scenarios and some TDD scenarios, the jointly optimized design significantly outperforms the fixed-uplink (fixed-downlink) heuristic of using the IRS configuration optimized for uplink (downlink) to assist downlink (uplink) transmissions. Moreover, the joint design substantially bridges the gap to the individual design upper bound of allowing different IRS configurations in uplink and downlink. Otherwise, in the remaining TDD scenarios, the fixed-uplink and fixed-downlink designs nearly achieve the individual design performance and substantially reduce overhead and/or complexity compared to the optimized joint design and individual design.