A (folklore?) conjecture states that no holomorphic modular form $F(τ)=\sum_{n=1}^\infty a_nq^n\in q\mathbb Z[[q]]$ exists, where $q=e^{2πiτ}$, such that its anti-derivative $\sum_{n=1}^\infty a_nq^n/n$ has integral coefficients in the $q$-expansion. A recent observation of Broadhurst and Zudilin, rigorously accomplished by Li and Neururer, led to examples of meromorphic modular forms possessing the integrality property. In this note we investigate the arithmetic phenomenon from a systematic perspective and discuss related transcendental extensions of the differentially closed ring of quasi-modular forms.