Let $p,q,l$ be three distinct prime numbers and let $N$ be a positive integer coprime to $pql$. For an integer $n\ge 0$, we define the directed graph $X_l^q(p^nN)$ whose vertices are given by isomorphism classes of elliptic curves over a finite field of characteristic $q$ equipped with a level $p^nN$ structure. The edges of $X_l^q(p^nN)$ are given by $l$-isogenies. We are interested in when the connected components of $X_l^q(p^nN)$ give rise to a tower of Galois covers as $n$ varies. We show that only in the supersingular case we do get a tower of Galois covers. We also study similar towers of isogeny graphs given by oriented supersingular curves, as introduced by Colò-Kohel, enhanced with a level structure.