We establish an isomorphism between the center of the twisted Heisenberg category and the subalgebra of the symmetric functions $Γ$ generated by odd power sums. We give a graphical description of the factorial Schur $Q$-functions as closed diagrams in the twisted Heisenberg category and show that the bubble generators of the center correspond to two sets of generators of $Γ$ which encode data related to up/down transition functions on the Schur graph. Finally, we describe an action of the trace of the twisted Heisenberg category, the $W$-algebra $W^-\subset W_{1+\infty}$, on $Γ$.