We study a discrete-time Markov process on triangular arrays of matrices of size $d\geq 1$, driven by inverse Wishart random matrices. The components of the right edge evolve as multiplicative random walks on positive definite matrices with one-sided interactions and can be viewed as a $d$-dimensional generalisation of log-gamma polymer partition functions. We establish intertwining relations to prove that, for suitable initial configurations of the triangular process, the bottom edge has an autonomous Markovian evolution with an explicit transition kernel. We then show that, for a special singular initial configuration, the fixed-time law of the bottom edge is a matrix Whittaker measure, which we define. To achieve this, we perform a Laplace approximation that requires solving a constrained minimisation problem for certain energy functions of matrix arguments on directed graphs.