The class of permutations that avoid the bivincular pattern (231, {1},{1}) is known to be enumerated by the Fishburn numbers. In this paper, we call them Fishburn permutations and study their pattern avoidance. For classical patterns of size 3, we give a complete enumerative picture for regular and indecomposable Fishburn permutations. For patterns of size 4, we focus on a Wilf equivalence class of Fishburn permutations that are enumerated by the Catalan numbers. In addition, we also discuss a class enumerated by the binomial transform of the Catalan numbers and give conjectures for other equivalence classes of pattern-avoiding Fishburn permutations.