A toric variety is constructed from a lattice polytope. It is common in algebraic combinatorics to carry this way a notion of an algebraic property from the variety to the polytope. From the combinatorial point of view, one of the most interesting constructions of toric varieties comes from the base polytope of a matroid. Matroid base polytopes and independence polytopes are Cohen--Macaulay. We study two natural stronger algebraic properties -- Gorenstein and smooth. We provide a full classifications of matroids whose independence polytope or base polytope is smooth or Gorenstein. The latter answers to a question raised by Herzog and Hibi.