The cognitive process of opinion formation is often characterized by stubbornness or resistance of agents to changes of opinion. To capture such a feature we introduce a constant latency time in the standard voter model of opinion dynamics: after switching opinion, an agent must keep it for a while. This seemingly simple modification drastically changes the stochastic diffusive behavior of the original model, leading to deterministic dynamical oscillations in the average opinion of the agents. We explain the origin of the oscillations and develop a mathematical formulation of the dynamics that is confirmed by extensive numerical simulations. We further characterize the rich phase space of the model and its asymptotic behavior. Our work offers insights into understanding and modeling opinion swings in diverse social contexts.