We investigate alternating sign matrices that are not permutation matrices, but have finite order in a general linear group. We classify all such examples of the form $P+T$, where $P$ is a permutation matrix and $T$ has four non-zero entries, forming a square with entries $1$ and $-1$ in each row and column. We show that the multiplicative orders of these matrices do not always coincide with those of permutation matrices of the same size. We pose the problem of identifying finite subgroups of general linear groups that are generated by alternating sign matrices.