Let $n$ be a positive integer, and let $A$ be a set of $k\ge 2n-1$ integers. For the restricted sumset $$ S_n(A)=\{a_1+\cdots +a_n:\ a_1,\ldots,a_n\in A,\ \text{and}\ a_i^2\neq a_j^2\ \text{for} \ 1\le i<j\le n\}, $$ by a 2002 result of Liu and Sun we have $$|S_n(A)|\ge (k-1)n-\frac 32n(n-1)+1.$$ In this paper, we determine the structure of $A$ when the lower bound is attained.