There are exactly two non-commutative rings of size $4$, namely, $E = \langle a, b ~\vert ~ 2a = 2b = 0, a^2 = a, b^2 = b, ab= a, ba = b\rangle$ and its opposite ring $F$. These rings are non-unital. A subset $D$ of $E^m$ is defined with the help of simplicial complexes, and utilized to construct linear left-$E$-codes $C^L_D=\{(v\cdot d)_{d\in D} : v\in E^m\}$ and right-$E$-codes $C^R_D=\{(d\cdot v)_{d\in D} : v\in E^m\}$. We study their corresponding binary codes obtained via a Gray map. The weight distributions of all these codes are computed. We achieve a couple of infinite families of optimal codes with respect to the Griesmer bound. Ashikhmin-Barg's condition for minimality of a linear code is satisfied by most of the binary codes we constructed here. All the binary codes in this article are few-weight codes, and self-orthogonal codes under certain mild conditions. This is the first attempt to study the structure of linear codes over non-unital non-commutative rings using simplicial complexes.