In this paper we construct Young wall models for the level $1$ highest weight and Fock space crystals of quantum affine algebras in types $E_6^{(2)}$ and $F_4^{(1)}$. Our starting point in each case is a combinatorial realization for a certain level $1$ perfect crystal in terms of Young columns. Then using energy functions and affine energy functions we define the notions of reduced and proper Young walls, which model the highest weight and Fock space crystals respectively.