We introduce a method for sparsifying distributed algorithms and exhibit how it leads to improvements that go past known barriers in two algorithmic settings of large-scale graph processing: Massively Parallel Computation (MPC), and Local Computation Algorithms (LCA). - MPC with Strongly Sublinear Memory: Recently, there has been growing interest in obtaining MPC algorithms that are faster than their classic $O(\log n)$-round parallel counterparts for problems such as MIS, Maximal Matching, 2-Approximation of Minimum Vertex Cover, and $(1+ε)$-Approximation of Maximum Matching. Currently, all such MPC algorithms require $\tildeΩ(n)$ memory per machine. Czumaj et al. [STOC'18] were the first to handle $\tildeΩ(n)$ memory, running in $O((\log\log n)^2)$ rounds. We obtain $\tilde{O}(\sqrt{\log Δ})$-round MPC algorithms for all these four problems that work even when each machine has memory $n^α$ for any constant $α\in (0, 1)$. Here, $Δ$ denotes the maximum degree. These are the first sublogarithmic-time algorithms for these problems that break the linear memory barrier. - LCAs with Query Complexity Below the Parnas-Ron Paradigm: Currently, the best known LCA for MIS has query complexity $Δ^{O(\log Δ)} poly(\log n)$, by Ghaffari [SODA'16]. As pointed out by Rubinfeld, obtaining a query complexity of $poly(Δ\log n)$ remains a central open question. Ghaffari's bound almost reaches a $Δ^{Ω\left(\frac{\log Δ}{\log\log Δ}\right)}$ barrier common to all known MIS LCAs, which simulate distributed algorithms by learning the local topology, à la Parnas-Ron [TCS'07]. This barrier follows from the $Ω(\frac{\log Δ}{\log\log Δ})$ distributed lower bound of Kuhn, et al. [JACM'16]. We break this barrier and obtain an MIS LCA with query complexity $Δ^{O(\log\log Δ)} poly(\log n)$.