Let G be a connected graph. The toughness of G is defined as t(G)=min{\frac{|S|}{c(G-S)}}, in which the minimum is taken over all proper subsets S\subset V(G) such that c(G-S)\geq 2 where c(G-S) denotes the number of components of G-S. Confirming a conjecture of Brouwer, Gu [SIAM J. Discrete Math. 35 (2021) 948--952] proved a tight lower bound on toughness of regular graphs in terms of the second largest absolute eigenvalue. Fan, Lin and Lu [European J. Combin. 110 (2023) 103701] then studied the toughness of simple graphs from the spectral radius perspective. While the toughness is an important concept in graph theory, it is also very interesting to study |S| for which c(G-S)\geq l for a given integer l\geq 2. This leads to the concept of the l-connectivity, which is defined to be the minimum number of vertices of G whose removal produces a disconnected graph with at least l components or a graph with fewer than l vertices. Gu [European J. Combin. 92 (2021) 103255] discovered a lower bound on the l-connectivity of regular graphs via the second largest absolute eigenvalue. As a counterpart, we discover the connection between the l-connectivity of simple graphs and the spectral radius. We also study similar problems for digraphs and an edge version.