In this paper, we associate a finite dimensional algebra, called a Brauer graph algebra, to every clean dessin d'enfant by constructing a quiver based on the monodromy of the dessin. We show that Galois conjugate dessins d'enfants give rise to derived equivalent Brauer graph algebras and that the stable Auslander-Reiten quiver and the dimension of the Brauer graph algebra are invariant under the induced action of the absolute Galois group.