Ascent sequences and their modified version play a central role in the bijective framework relating several combinatorial structures counted by the Fishburn numbers. Ascent sequences are positive integer sequences defined by imposing a bound on the growth of their entries in terms of the number of ascents contained in the corresponding prefix, while modified ascent sequences are the image of ascent sequences under the so-called hat map. By relaxing the notion of ascent, Dukes and Sagan have recently introduced difference ascent sequences. Here we define modified difference ascent sequences and study their combinatorial properties. Inversion sequences are a superset of the difference ascent sequences and we extend the hat map to this domain. Our extension depends on a parameter which we specialize to obtain a new set of permutations counted by the Fishburn numbers and characterized by a subdiagonality property.