Similar to how standard Young tableaux represent paths in the Young lattice, Latin rectangles may be use to enumerate paths in the poset of semi-magic squares with entries zero or one. The symmetries associated to determinant preserve this poset, and we completely describe the orbits, covering data, and maximal chains for squares of size 4, 5, and 6. The last item gives the number of Latin squares in these cases. To calculate efficiently for size 6, we in turn identify orbits with certain equivalence classes of hypergraphs.