In analogy with the origin of the additive structure of $K$-theory, we construct an $E_\infty$ structure on the matroid Grassmannian (the space of oriented matroids) for which the underlying binary operation is the direct sum of matroids. The proof involves lifting the polyhedral fan structure of the Dressian to a polyhedral model for the matroid Grassmannian, and introducing a novel $E_\infty$ operad made from the space of infinite subsets of $\mathbb{N}$.