We show an algorithm for computing the permanent of a random matrix with vanishing mean in quasi-polynomial time. Among special cases are the Gaussian, and biased-Bernoulli random matrices with mean 1/lnln(n)^{1/8}. In addition, we can compute the permanent of a random matrix with mean 1/poly(ln(n)) in time 2^{O(n^{\eps})} for any small constant \eps>0. Our algorithm counters the intuition that the permanent is hard because of the "sign problem" - namely the interference between entries of a matrix with different signs. A major open question then remains whether one can provide an efficient algorithm for random matrices of mean 1/poly(n), whose conjectured #P-hardness is one of the baseline assumptions of the BosonSampling paradigm.