For $m,n\in\mathbb{N}$ let $X=(X_{ij})_{i\leq m,j\leq n}$ be a random matrix, $A=(a_{ij})_{i\leq m,j\leq n}$ a real deterministic matrix, and $X_A=(a_{ij}X_{ij})_{i\leq m,j\leq n}$ the corresponding structured random matrix. We study the expected operator norm of $X_A$ considered as a random operator between $\ell_p^n$ and $\ell_q^m$ for $1\leq p,q \leq \infty$. We prove optimal bounds up to logarithmic terms when the underlying random matrix $X$ has i.i.d. Gaussian entries, independent mean-zero bounded entries, or independent mean-zero $ψ_r$ ($r\in(0,2]$) entries. In certain cases, we determine the precise order of the expected norm up to constants. Our results are expressed through a sum of operator norms of Hadamard products $A\circ A$ and $(A\circ A)^T$.