Recently, reconfigurable intelligent surface (RIS) has immensely been deployed to overcome blockage issue and widen coverage for enabling superior performance 6G networks. Mainly, systems use RIS as an assistant to redirect the transmitter (TX) incident signal towards the receiver (RX) by configuring RIS elements amplitudes and phase shifts in a passive beamforming (PBF) process. Channel estimation (CE) based PBF schemes achieve optimal performance, but they need high overhead and time consumption, especially with high number of RIS elements. Codebook (CB) based PBF solutions can be alternatives to overcome these issues by only searching through a limited reflection patterns (RPs) and determining the optimal one based on a predefined metric. However, they consume high power and time relevant to the used CB size. In this work, we propose a direction based PBF (D-PBF) scheme, where we aim to map between the RXs directions and the codebook RPs and store this information in an updated database (DB). Hence, if the matching between a coming RX and a particular RP exists, the proposed scheme will directly select this RP to configure the RIS elements, otherwise, it memorizes this codeword for future searching. Finally, if the matching failed, searching through the memorized RPs will be done to find the optimal one, then updating the DB accordingly. After a time period, which depends on the CB size, the DB will converge, and the D-PBF scheme will need no searching to select the optimal RP. Hence, the proposed scheme needs extremely lower overhead, power, and time comparable to the CE and conventional CB based solutions, while obtaining acceptable performance in terms of effective rate.