This paper considers submodular function minimization (SFM) restricted to a family of subsets. We show that SFM over complements of families with certain hierarchical structures can be solved in polynomial-time. This yields a polynomial-time algorithm for SFM over complements of various families, such as intersecting families, crossing families, and the unions of lattices. Moreover, this tractability result partially settles the open question posed by Nägele, Sudakov, and Zenklusen on polynomial-solvability of SFM over the intersection of parity families. Furthermore, our tractability result implies that for a constant positive integer $k$, the $k$-th smallest value of a submodular function can be obtained in polynomial-time.