Let $Γ$ be the graph whose vertices are the chambers of the finite projective space $PG(3,q)$ with two vertices being adjacent when the corresponding chambers are in general position. It is known that the independence number of this graph is $(q^2+q+1)(q+1)^2$. For $q\ge 43$ we determine the largest independent set of $Γ$ and show that every maximal independent set that is not a largest one has at most constant times $q^3$ elements. For $q\ge 47$, this information is then used to show that $Γ$ has chromatic number $q^2+q$. Furthermore, for many families of generalized quadrangles we prove similar results for the graph that is built in the same way on the chambers of the generalized quadrangle.