This paper investigates the physical layer security issue of a device-to-device (D2D) underlaid cellular system with a multi-antenna base station (BS) and a multi-antenna eavesdropper. To investigate the potential of D2D communication in improving network security, the conventional network without D2D users (DUs) is first considered. It is shown that the problem of maximizing the sum secrecy rate (SR) of cellular users (CUs) for this special case can be transformed to an assignment problem and optimally solved. Then, a D2D underlaid network is considered. Since the joint optimization of resource block (RB) allocation, CU-DU matching and power control is a mixed integer programming, the problem is difficult to handle. Hence, the RB assignment process is first conducted by ignoring D2D communication, and an iterative algorithm is then proposed to solve the remaining problem. Simulation results show that the sum SR of CUs can be greatly increased by D2D communication, and compared with the existing schemes, a better secrecy performance can be obtained by the proposed algorithms.