The set of finite binary matrices of a given size is known to carry a finite type A bicrystal structure. We first review this classical construction, explain how it yields a short proof of the equality between Kostka polynomials and one-dimensional sums together with a natural generalisation of the 2M -- X Pitman transform. Next, we show that, once the relevant formalism on families of infinite binary matrices is introduced, this is a particular case of a much more general phenomenon. Each such family of matrices is proved to be endowed with Kac-Moody bicrystal and tricrystal structures defined from the classical root systems. Moreover, we give an explicit decomposition of these multicrystals, reminiscent of the decomposition of characters yielding the Cauchy identities.