Assessment of voice signals has long been performed with the assumption of periodicity as this facilitates analysis. Near periodicity of normal voice signals makes short-time harmonic modeling an appealing choice to extract vocal feature parameters. For dysphonic voice, however, a fixed harmonic structure could be too constrained as it strictly enforces periodicity in the model. Slight variation in amplitude or frequency in the signal may cause the model to misrepresent the observed signal. To address these issues, this paper presents a time-varying harmonic model, which allows its fundamental frequency and harmonic amplitudes to be polynomial functions of time. The model decouples the slow deviations of frequency and amplitude from fast irregular vocal fold vibratory behaviors such as subharmonics and diplophonia. The time-varying model is shown to track the frequency and amplitude modulations present in voice with severe tremor. This reduces the sensitivity of the model-based harmonics-to-noise ratio measures to slow frequency and amplitude variations while maintaining its sensitivity to increase in turbulent noise or the presence of irregular vibration. Other uses of the model include the vocal tract filter estimation and the rates of frequency and intensity changes. These use cases are experimentally demonstrated along with the modeling accuracy.