We study the asymptotic behavior of cumulants of lacunary trigonometric sums $S_n(ω) := \sum_{k=1}^n \cos (2 πa_k ω)$, $ω\in[0,1]$, and show that cumulant growth is highly sensitive to the arithmetic structure of the sequence $(a_k)_{k \geq 1}$ of positive integers. In particular, if $\lim_{k \to \infty} a_{k+1}/a_k = η> 1$ for some transcendental number $η$, we prove that for every $m\in \mathbb N$ the $m$-th cumulant of $S_n$ is asymptotically equivalent to the $m$-th cumulant of the ``independent model'' $\widetilde{S}_n := \sum_{k=1}^n \cos (2 πa_k U_k)$, where $U_1, U_2, \dots$ are independent random variables having uniform distribution on $[0,1]$. In particular, the order of growth of the cumulants as $n \to \infty$ is linear in this case. We also show that the transcendence condition for $\lim_{k \to \infty} a_{k+1}/a_k$ is in general necessary: when the ratio limit $η$ is algebraic, the cumulants of $S_n$ may have a different asymptotic order from those of $\widetilde{S}_n$. For instance, for $a_k = 2^k+1$ (with $η= 2$), the sixth cumulant of $S_n$ grows quadratically in $n$. In contrast, for $a_k = 2^k$ (again $η= 2$) or when $(a_k)_{k \geq 1}$ is the Fibonacci sequence (with $η= (1+\sqrt 5)/2$), the $m$-th cumulant of $S_n$ grows linearly as $n\to\infty$, but with a growth rate that differs from the one of the independent model $\widetilde{S}_n$. Overall, our results show that the asymptotic behavior of the cumulants of lacunary trigonometric sums depends on arithmetic effects in a very delicate way. This is particularly remarkable since many other probabilistic limit theorems, such as the Central Limit Theorem, hold for lacunary trigonometric sums in a universal way without any such sensitivity towards arithmetic effects.