The Terwilliger algebras of association schemes over an arbitrary field $\mathbb{F}$ were called the Terwilliger $\mathbb{F}$-algebras of association schemes in [9]. In this paper, we study the Terwilliger $\mathbb{F}$-algebras of factorial association schemes. We determine the centers, the semisimplicity, the Jacobson radicals and their nilpotent indices, the Wedderburn-Artin decompositions of the Terwilliger $\mathbb{F}$-algebras of factorial association schemes. Moreover, we determine all Terwilliger $\mathbb{F}$-algebras of factorial association schemes that are the symmetric $\mathbb{F}$-algebras or the Frobenius $\mathbb{F}$-algebras.