A conjecture of Kalai from 1994 posits that for an arbitrary $2\leq k\leq \lfloor d/2 \rfloor$, the combinatorial type of a simplicial $d$-polytope $P$ is uniquely determined by the $(k-1)$-skeleton of $P$ (given as an abstract simplicial complex) together with the space of affine $k$-stresses on $P$. We establish the first non-trivial case of this conjecture, namely, the case of $k=2$. We also prove that for a general $k$, Kalai's conjecture holds for the class of $k$-neighborly polytopes.