Polarization reconfigurable (PR) antennas enhance spectrum and energy efficiency between next-generation node B(gNB) and user equipment (UE). This is achieved by tuning the polarization vectors for each antenna element based on channel state information (CSI). On the other hand, degree of freedom increased by PR antennas yields a challenge in channel estimation with pilot training overhead. This paper pursues the reduction of pilot overhead, and proposes to employ deep neural networks (DNNs) on both transceiver ends to directly optimize the polarization and beamforming vectors based on the received pilots without the explicit channel estimation. Numerical experiments show that the proposed method significantly outperforms the conventional first-estimate-then-optimize scheme by maximum of 20% in beamforming gain.