Free Fermions on vertices of distance-regular graphs are considered. Bipartition are defined by taking as one part all vertices at a given distance from a reference vertex. The ground state is constructed by filling all states below a certain energy. Borrowing concepts from time and band limiting problems, algebraic Heun operators and Terwilliger algebras, it is shown how to obtain, quite generally, a block tridiagonal matrix that commutes with the entanglement Hamiltonian. The case of the Hadamard graphs is studied in details within that framework and the existence of the commuting matrix is shown to allow for an analytic diagonalization of the restricted two-point correlation matrix and hence for an explicit determination of the entanglement entropy.