There are some good combinatorial structures suitable for image encryption. In this study, a new chaotic image encryption algorithm based on transversals in a Latin square is proposed. By means of an n-transversal of a Latin square, we can permutate an image data group by group for the first time, then use two Latin squares for auxiliary diffusion on the basis of a chaotic sequence, finally make use of a pair of orthogonal Latin squares to do the second scrambling. As a whole, the encryption process of "scrambling-diffusion-scrambling" is formed. The experimental results indicate that this algorithm achieves a secure and fast encryption effect. The final information entropy is very close to 8, and the correlation coefficient is approximate to 0. All these tests verify the robustness and practicability of this proposed algorithm.