We introduce the $B$-Stirling numbers of the first and second kind, which are the coefficients of the potential polynomials when we express them in terms of the monomials and the falling factorials, respectively. These numbers include, as particular cases, the partial and complete Bell polynomials, the degenerate and probabilistic Stirling numbers, and the $S$-restricted Stirling numbers, among others. Special attention is devoted to the computation of such numbers. On the one hand, a recursive formula is provided. On the other, we can compute Stirling numbers of one kind in terms of the other, with the help of the classical Stirling numbers.