Reconfigurable Intelligent Surfaces (RIS) is a transformative technology with great potential in many applications in wireless communications and realizing the Internet of Everything at sixth generation (6G). In this study, we propose a wireless system where the RIS acts as an antenna, which we call Reconfigurable Intelligent Metasurface Antennas (RIMSA). In particular, the base station (BS) equipped with a RIMSA array performs downlink transmissions to multiple users, where each user has a single or multiple RIMSA/RF links, and we aim to solve the sum-rate maximization problem by jointly optimizing the digital processing matrix of the transceivers and the phase responses of RIMSA array at both BS and users. For the multi-user multiple-input single-output (MU-MISO) scenario, we develop an alternating optimization algorithm to slove the problem, where a fractional programming (FP) is used to optimize the digital processing matrix and a product manifold optimization (PMO) is proposed to provide the optimal phase responses of the RIMSA array at both BS and users. For the multi-user multiple-input multiple-output (MU-MIMO) scenario, we equate it to a weighted sum of mean square errors minimization problem, which can be solved by three subproblems iteratively. Both the optimal digital precoder subproblem and the optimal digital combiner subproblem have closed-form solutions, and the subproblem of RIMSA configuration is solved by the PMO algorithm as well. Simulation results demonstrate that the proposed algorithms achieve significant performance gains over conventional algorithms.