In continuation of our recent work arXiv:2006.07312, we classify the extremal traces on infinite diagram algebras that appear in the context of Schur-Weyl duality for Banica and Speicher's easy groups. We show that the branching graphs of these algebras describe walks on new variations of the Young graph which describe curious ways of growing Young diagrams. As a consequence, we prove that the extremal traces on generic rook-Brauer algebras are always extensions of extremal traces on the group algebra $\mathbb{C}[S_{\infty}]$ of the infinite symmetric group. Moreover, we conjecture that the same is true for generic parameter deformations of the centralizers of the hyperoctahedral group and we reduce this conjecture to a conceptually much simpler numerical statement. Lastly, we address the trace classification problem for the Schur-Weyl dual of the halfliberated orthogonal group $O_N^*$, in which case extremal traces are always extensions of extremal traces on $\mathbb{C}[S_{\infty} \times S_{\infty}]$. Our approach relies on methods developed by Vershik and Nikitin.