Given a configuration $A$ of $n$ points in $\mathbb{R}^{d-1}$, we introduce the higher secondary polytopes $Σ_{A,1},\dots, Σ_{A,n-d}$, which have the property that $Σ_{A,1}$ agrees with the secondary polytope of Gelfand--Kapranov--Zelevinsky, while the Minkowski sum of these polytopes agrees with Billera--Sturmfels' fiber zonotope associated with (a lift of) $A$. In a special case when $d=3$, we refer to our polytopes as higher associahedra. They turn out to be related to the theory of total positivity, specifically, to certain combinatorial objects called plabic graphs, introduced by the second author in his study of the totally positive Grassmannian. We define a subclass of regular plabic graphs and show that they correspond to the vertices of the higher associahedron $Σ_{A,k}$, while square moves connecting them correspond to the edges of $Σ_{A,k}$. Finally we connect our polytopes to soliton graphs, the contour plots of soliton solutions to the KP equation, which were recently studied by Kodama and the third author. In particular, we confirm their conjecture that when the higher times evolve, soliton graphs change according to the moves for plabic graphs.