We study the properties of the score confidence set for the local average treatment effect in non and semiparametric instrumental variable models. This confidence set is constructed by inverting a score test based on an estimate of the nonparametric influence function for the estimand, and is known to be uniformly valid in models that allow for arbitrarily weak instruments; because of this, the confidence set can have infinite diameter at some laws. We characterize the six possible forms the score confidence set can take: a finite interval, an infinite interval (or a union of them), the whole real line, an empty set, or a single point. Moreover, we show that, at any fixed law, the score confidence set asymptotically coincides, up to a term of order 1/n, with the Wald confidence interval based on the doubly robust estimator which solves the estimating equation associated with the nonparametric influence function. This result implies that, in models where the efficient influence function coincides with the nonparametric influence function, the score confidence set is, in a sense, optimal in terms of its diameter. We also show that under weak instrument asymptotics, where the strength of the instrument is modelled as local to zero, the doubly robust estimator is asymptotically biased and does not follow a normal distribution. A simulation study confirms that, as expected, the doubly robust estimator performs poorly when instruments are weak, whereas the score confidence set retains good finite-sample properties in both strong and weak instrument settings. Finally, we provide an algorithm to compute the score confidence set, which is now available in the DoubleML package for double machine learning.