In this article, we studied the inverse Erdős-Heilbronn problem with the restricted sumset from two components $A$ and $B$ that are not necessarily the same. We give a completely elementary proof for the problem in $\mathbb{Z}$ and some partial results that contributes to the elementary proof of the problem in $\mathbb{Z}/p\mathbb{Z}$, avoiding the usage of the powerful polynomial method and the Combinatorial Nullstellensatz.