Bicoherence analysis is a well established method for identifying the quadratic nonlinearity of stationary processes. However, it is often applied without checking the basic assumptions of stationarity and convergence. The classic bicoherence, unfortunately, tends to give false positives -- high bicoherence values without actual nonlinear coupling of different frequency components -- for signals exhibiting rapidly changing amplitudes and limited length. The effect of false positive values can lead to misinterpretation of results, therefore a more prudent analysis is necessary in such cases. This paper analyses the properties of bispectrum and bicoherence in detail, generalizing these quantities to nonstationary processes. A step-by-step method is proposed to filter out false positives at a given confidence level for the case of nonstationary signals. We present a number of test cases, where the method is demonstrated on simple physics-based numerical systems. The approach and methodology introduced in the paper can be generalized to lower and higher order coherence calculations.