A set of integers $A$ is non-averaging if there is no element $a$ in $A$ which can be written as an average of a subset of $A$ not containing $a$. We show that the largest non-averaging subset of $\{1, \ldots, n\}$ has size $n^{1/4+o(1)}$, thus solving the Erdős-Straus problem. We also determine the largest size of a non-averaging set in a $d$-dimensional box for any fixed $d$. Our main tool includes the structure theorem for the set of subset sums due to Conlon, Fox and the first author, together with a result about the structure of a point set in nearly convex position.