We consider three different models of sparse random graphs:~undirected and directed Erdős-Rényi graphs, and random bipartite graph with an equal number of left and right vertices. For such graphs we show that if the edge connectivity probability $p \in (0,1)$ satisfies $n p \ge \log n + k(n)$ with $k(n) \to \infty$ as $n \to \infty$, then the adjacency matrix is invertible with probability approaching one (here $n$ is the number of vertices in the two former cases and the number of left and right vertices in the latter case). If $np \le \log n -k(n)$ then these matrices are invertible with probability approaching zero, as $n \to \infty$. In the intermediate region, when $np=\log n + k(n)$, for a bounded sequence $k(n) \in \mathbb{R}$, the event $Ω_0$ that the adjacency matrix has a zero row or a column and its complement both have non-vanishing probability. For such choices of $p$ our results show that conditioned on the event $Ω_0^c$ the matrices are again invertible with probability tending to one. This shows that the primary reason for the non-invertibility of such matrices is the existence of a zero row or a column. The bounds on the probability of the invertibility of these matrices are a consequence of quantitative lower bounds on their smallest singular values. Combining this with an upper bound on the largest singular value of the centered version of these matrices we show that the (modified) condition number is $O(n^{1+o(1)})$ on the event that there is no zero row or column, with large probability. This matches with von Neumann's prediction about the condition number of random matrices up to a factor of $n^{o(1)}$, for the entire range of $p$.