We prove that there is a set of integers $A$ having positive upper Banach density whose difference set $A-A:=\{a-b:a,b\in A\}$ does not contain a Bohr neighborhood of any integer, answering a question asked by Bergelson, Hegyvári, Ruzsa, and the author, in various combinations. In the language of dynamical systems, this result shows that there is a set of integers $S$ which is dense in the Bohr topology of $\mathbb Z$ and which is not a set of measurable recurrence. Our proof yields the following stronger result: if $S\subseteq \mathbb Z$ is dense in the Bohr topology of $\mathbb Z$, then there is a set $S'\subseteq S$ such that $S'$ is dense in the Bohr topology of $\mathbb Z$ and for all $m\in \mathbb Z,$ the set $(S'-m)\setminus \{0\}$ is not a set of measurable recurrence.