In 2011, Duncan and Steingrímsson conjectured that modified ascent sequences avoiding any of the patterns 212, 1212, 2132, 2213, 2231 and 2321 are counted by the Bell numbers. Furthermore, the distribution of the number of ascents is the reverse of the distribution of blocks on set partitions. We solve the conjecture for all the patterns except 2321. We describe the corresponding sets of Fishburn permutations by pattern avoidance, and leave some open questions for future work.