Let $Ω_i$ and $Ω_j$ be the sets of elements of respective types $i$ and $j$ of a polar space~$Δ$ of rank at least $3$, viewed as a Tits-building. For any Weyl distance $δ$ between $Ω_i$ and $Ω_j$, we show that $δ$ is characterised by $i$ and $j$ and two additional numerical parameters $k$ and $\ell$. We consider permutations $ρ$ of $Ω_i \cup Ω_j$ that preserve a single Weyl distance $δ$. Up to a minor technical condition on $\ell$, we prove that, up to trivial cases and two classes of true exceptions, $ρ$ is induced by an automorphism of the Tits-building associated to $Δ$, which is always a type-preserving automorphism of $Δ$ (and hence preserving all Weyl-distances), unless $Δ$ is hyperbolic, in which case there are outer automorphisms. For each class of exceptions, we determine a Tits-building $Δ'$ in which $Δ$ naturally embeds and is such that $ρ$ is induced by an automorphism of $Δ'$. At the same time, we prove similar results for permutations preserving a natural incidence condition. These yield combinatorial characterisations of all groups of algebraic origin which are the full automorphism group of some polar space as the automorphism group of many bipartite graphs.