REST (Representational State Transfer) APIs have become integral for data communication and exchange due to their simplicity, scalability, and compatibility with web standards. However, ensuring REST APIs' reliability through rigorous testing poses significant challenges, given the complexities of operations, parameters, inputs, dependencies, and call sequences. In this paper, we introduce MUCOREST, a novel Reinforcement Learning (RL)-based API testing approach that leverages Q-learning to maximize code coverage and output coverage, thereby improving bug discovery. By focusing on these proximate objectives rather than the abstract goal of maximizing failures, MUCOREST effectively discovers critical code areas and diverse API behaviors. The experimental results on a benchmark of 10 services show that MUCOREST significantly outperforms state-of-the-art API testing approaches by 11.6-261.1% in the number of discovered API bugs. MUCOREST can generate much fewer API calls to discover the same number of bugs compared to the other approaches. Furthermore, 12.17%-64.09% of the bugs discovered by the other techniques can also be found by MUCOREST.