Simultaneous wireless information and power transfer (SWIPT) is anticipated to have great applications in fifth-generation (5G) and beyond communication systems. In this paper, we address the energy efficiency (EE) optimization problem for SWIPT multiple-input multiple-output broadcast channel (MIMO-BC) with time-switching (TS) receiver design. Our aim is to maximize the EE of the system whilst satisfying certain constraints in terms of maximum transmit power and minimum harvested energy per user. The coupling of the optimization variables, namely, transmit covariance matrices and TS ratios, leads to a EE problem which is non-convex, and hence very difficult to solve directly. Hence, we transform the original maximization problem with multiple constraints into a min-max problem with a single constraint and multiple auxiliary variables. We propose a dual inner/outer layer resource allocation framework to tackle the problem. For the inner-layer, we invoke an extended SWIPT-based BC-multiple access channel (MAC) duality approach and provide two iterative resource allocation schemes under fixed auxiliary variables for solving the dual MAC problem. A sub-gradient searching scheme is then proposed for the outer-layer in order to obtain the optimal auxiliary variables. Numerical results confirm the effectiveness of the proposed algorithms and illustrate that significant performance gain in terms of EE can be achieved by adopting the proposed extended BC-MAC duality-based algorithm.