We introduce the notion of $p$-adic asymptotics, or $p$-asymptotics, to the context of finite-index subgroup and subalgebra enumeration. For finitely generated groups and finite-dimensional algebras, we connect these asymptotics with the poles of their associated local zeta functions. Our two main results establish the smallest real pole for local zeta functions associated with residually nilpotent algebras, as well as its simplicity and residue whenever this algebra is graded. We thereby provide proof to parts of two conjectures raised by Rossmann and give a precise description of the $p$-asymptotic behaviour inside these algebras.