We describe here how the recent Wagner's approach for applying reinforcement learning to construct examples in graph theory can be used in the search for critical graphs for small Ramsey numbers. We illustrate this application by providing lower bounds for the small Ramsey numbers $R(K_{2,5}, K_{3,5})$, $R(B_3, B_6)$ and $R(B_4, B_5)$ and by improving the lower known bound for $R(W_5, W_7)$.