Let $A$ be a finite set of $k$ integers. For $h \leq k$, the restricted $h$-fold sumset $h^{\wedge} A$ is the set of all sums of $h$ distinct elements of $A$. In additive combinatorics, much of the focus has traditionally been on finite integer sets whose sumsets are unusually small (cf.\ Freiman's theorem and its extensions). More recently, Nathanson posed the inverse problem for the restricted sumset $h^{\wedge} A$ when $|h^{\wedge} A|$ is small. For $h \in \{2, 3, 4\}$, this question has already been studied by Mohan and Pandey. In this article, we study the inverse problems for $h^{\wedge} A$ with arbitrary $h \geq 3$ and characterize all possible sets $A$ for certain cardinalities of $h^{\wedge} A$.