Let $K \subset \R^d$ be a smooth convex set and let $¶_\la$ be a Poisson point process on $\R^d$ of intensity $\la$. The convex hull of $¶_\la \cap K$ is a random convex polytope $K_\la$. As $\la \to \infty$, we show that the variance of the number of $k$-dimensional faces of $K_\la$, when properly scaled, converges to a scalar multiple of the affine surface area of $K$. Similar asymptotics hold for the variance of the number of $k$-dimensional faces for the convex hull of a binomial process in $K$.