This paper studies the symbol error rate performance of cognitive radio transmissions in the presence of imperfect sensing decisions. Two different transmission schemes, namely sensing-based spectrum sharing (SSS) and opportunistic spectrum access (OSA), are considered. In both schemes, secondary users first perform channel sensing, albeit with possible errors. In SSS, depending on the sensing decisions, they adapt the transmission power level and coexist with primary users in the channel. On the other hand, in OSA, secondary users are allowed to transmit only when the primary user activity is not detected. Initially, for both transmission schemes, general formulations for the optimal decision rule and error probabilities are provided for arbitrary modulation schemes under the assumptions that the receiver is equipped with the sensing decision and perfect knowledge of the channel fading, and the primary user's received faded signals at the secondary receiver has a Gaussian mixture distribution. Subsequently, the general approach is specialized to rectangular quadrature amplitude modulation (QAM). More specifically, optimal decision rule is characterized for rectangular QAM, and closed-form expressions for the average symbol error probability attained with the optimal detector are derived under both transmit power and interference constraints. The effects of imperfect channel sensing decisions, interference from the primary user and its Gaussian mixture model, and the transmit power and interference constraints on the error rate performance of cognitive transmissions are analyzed.