In this paper, we consider device-to-device (D2D) communication underlaying uplink cellular networks with multiple base stations (BSs), where each user can switch between traditional cellular mode (through BS) and D2D mode (by connecting proximity user), namely mode selection. We impose load balancing constraints on BSs to efficient resource usage. The joint problem of mode selection and user association is formulated as a combinatorial problem and NP-complete. We adopt a graph-based approach to solve the problem globally optimally in polynomial time. To further reduce complexity, we also propose a distributed algorithm based on dual method. We show that the proposed distributed algorithm achieves nearly the same performance as the proposed optimal graph based algorithm.