Orthogonal and quasi-orthogonal matrices have a long history of use in digital image processing, digital and wireless communications, cryptography and many other areas of computer science and coding theory. The practical benefits of using orthogonal matrices come from the fact that the computation of inverse matrices is avoided, by simply using the transpose of the orthogonal matrix. In this paper, we introduce a new family of matrices over finite fields that we call \emph{Quasi-Binary and Quasi-Orthogonal Matrices}. We call the matrices quasi-binary due to the fact that matrices have only two elements $a, b \in \mathbb{F}_q$, but those elements are not $0$ and $1$. In addition, the reason why we call them quasi-orthogonal is due to the fact that their inverses are obtained not just by a simple transposition, but there is a need for an additional operation: a replacement of $a$ and $b$ by two other values $c$ and $d$. We give a simple relation between the values $a, b, c$ and $d$ for any finite field and especially for finite fields with characteristic 2. Our construction is based on incident matrices from cyclic Latin Rectangles and the efficiency of the proposed algorithm comes from the avoidance of matrix-matrix or matrix-vector multiplications.