The fifth generation of wireless networks (5G) necessitates the use of waveforms with loose constraints on synchronization in multiuser scenarios. Also, carrier aggregation, as a way to better utilize the spectrum in 5G, needs a waveform with low out-of-band (OOB) emission. Generalized frequency division multiplexing (GFDM) and circular filter bank multicarrier (C-FBMC) are two candidate waveforms that fulfill these requirements. Both GFDM and C-FBMC operate based on circular convolution, and use cyclic prefix to combat channel response. In this paper, we develop an analytical technique for examining the OOB emission and multiuser interference (MUI) in circularly shaped waveforms, like GFDM and C-FBMC. To stay focused, the study in this paper is limited to C-FBMC modulation. However, the approach we take is trivially extendable to other waveforms as well. We derive equations that quantify OOB emission and MUI. Our analysis allows us to identify the source of OOB emission and MUI. This leads us to quantify the methods proposed by other researchers to decrease OOB emission and MUI. Moreover, we quantify the impact of signal windowing at the transmitter and receiver in reducing OOB emission and MUI, respectively.