The Robinson-Schensted-Knuth (RSK) algorithm maps an integer matrix to a pair of semi-standard Young tableaux (SSYTs) whose underlying shape has the same integer partition. We study the set of matrices associated with a given partition $λ$ vis-a-vis the number of inversions of the matrix. In the case where the integer matrix is a permutation matrix, the resulting tableaux are standard Young tableaux or SYTs. Han (EJC, 2005) combinatorially studied the set of permutations that map to SYTs of shape $λ$ under the RSK algorithm and counted the permutations with the minimum number of inversions in that set, as well as formulated the minimal number of inversions. Han's work can be extended to a case where the matrix is a general integer matrix and the tableaux are semi-standard Young tableaux. We have conjectured a formula for the minimal number of inversions in the set of matrices with a fixed shape $λ$. We further provide a conjecture for the characterisation of the minimal generalised matrices.