Let $\mathbb{F}_q$ be a finite field of order $q$ and $\mathcal{E}$ be a set in $\mathbb{F}_q^d$. The distance set of $\mathcal{E}$, denoted by $Δ(\mathcal{E})$, is the set of distinct distances determined by the pairs of points in $\mathcal{E}$. Very recently, Iosevich, Koh, and Parshall (2018) proved that if $|\mathcal{E}|\gg q^{d/2}$, then the quotient set of $Δ(\mathcal{E})$ satisfies \[\left\vert\frac{Δ(\mathcal{E})}{Δ(\mathcal{E})}\right\vert=\left\vert \left\lbrace\frac{a}{b}\colon a, b\in Δ(\mathcal{E}), b\ne 0\right\rbrace\right\vert\gg q.\] In this paper, we break the exponent $d/2$ when $\mathcal{E}$ is a Cartesian product of sets over a prime field. More precisely, let $p$ be a prime and $A\subset \mathbb{F}_p$. If $\mathcal{E}=A^d\subset \mathbb{F}_p^d$ and $|\mathcal{E}|\gg p^{\frac{d}{2}-\varepsilon}$ for some $\varepsilon>0$, then we have \[\left\vert\frac{Δ(\mathcal{E})}{Δ(\mathcal{E})}\right\vert, ~\left\vert Δ(\mathcal{E})\cdot Δ(\mathcal{E})\right\vert \gg p.\] Such improvements are not possible over arbitrary finite fields. These results give us a better understanding about the structure of distance sets and the Erdős-Falconer distance conjecture over finite fields.