We investigate the large $N$ limit of permutation orbifolds of vertex operator algebras. To this end, we introduce the notion of nested oligomorphic permutation orbifolds and discuss under which conditions their fixed point VOAs converge. We show that if this limit exists, then it has the structure of a vertex algebra. Finally, we give an example based on $\mathrm{GL}(N,q)$ for which the fixed point VOA limit is also the limit of the full permutation orbifold VOA.