In this paper, we investigate the problem of the joint transceiver and offset design (JTOD) for point-to-point multiple-input-multiple-output (MIMO) and multiple user multiple-input-single-output (MU-MISO) visible light communication (VLC) systems. Both uplink and downlink multi-user scenarios are considered. The shot noise induced by the incoming signals is considered, leading to a more realistic MIMO VLC channel model. Under key lighting constraints, we formulate non-convex optimization problems aiming at minimizing the sum mean squared error. To optimize the transceiver and the offset jointly, a gradient projection based procedure is resorted to. When only imperfect channel state information is available, a semidefinite programming (SDP) based scheme is proposed to obtain robust transceiver and offset. The proposed method is shown to non-trivially outperform the conventional scaled zero forcing (ZF) and singular value decomposition (SVD) based equalization methods. The robust scheme works particularly well when the signal is much stronger than the noise.