We consider the problem of estimating the structural function in nonparametric instrumental regression, where in the presence of an instrument W a response Y is modeled in dependence of an endogenous explanatory variable Z. The proposed estimator is based on dimension reduction and additional thresholding. The minimax optimal rate of convergence of the estimator is derived assuming that the structural function belongs to some ellipsoids which are in a certain sense linked to the conditional expectation operator of Z given W. We illustrate these results by considering classical smoothness assumptions. However, the proposed estimator requires an optimal choice of a dimension parameter depending on certain characteristics of the unknown structural function and the conditional expectation operator of Z given W, which are not known in practice. The main issue addressed in our work is an adaptive choice of this dimension parameter using a model selection approach under the restriction that the conditional expectation operator of Z given W is smoothing in a certain sense. In this situation we develop a penalized minimum contrast estimator with randomized penalty and collection of models. We show that this data-driven estimator can attain the lower risk bound up to a constant over a wide range of smoothness classes for the structural function.