Ascent sequences form a central class of combinatorial objects, as they are in bijection with several important families such as (2+2)-free posets, Stoimenow matchings, and other Fishburn objects, and are enumerated by the Fishburn numbers. We study pattern avoidance in ascent sequences for the five patterns of length 4: $0101$, $0102$, $0112$, $0120$, and $0121$. These patterns arise naturally from recent work on pattern avoidance in related families of Fishburn objects, including Stoimenow matchings and (2+2)-free posets. We enumerate ascent sequences avoiding any subset of these patterns, with the exception of the sets $\{0120\}$, $\{0121\}$, and $\{0120,0121\}$, for which the enumeration remains open. Our results reveal that the corresponding avoidance classes fall into $16$ Wilf equivalence classes and exhibit a wide range of enumerative behaviour, including connections to classical sequences such as the Catalan and Fibonacci numbers, as well as polynomial formulas and rational generating functions; several of the sequences we obtain appear to be new. Our methods combine structural decompositions with generating-tree techniques and, in several cases, rely on reductions to shorter patterns via restricted growth functions. This work contributes to the broader study of pattern avoidance across Fishburn families and highlights further connections between ascent sequences and other combinatorial structures.