Wisdom of crowds refers to the phenomenon that the aggregate prediction or forecast of a group of individuals can be surprisingly more accurate than most individuals in the group, and sometimes - than any of the individuals comprising it. This article models the impact of social influence on the wisdom of crowds. We build a minimalistic representation of individuals as Brownian particles coupled by means of social influence. We demonstrate that the model can reproduce results of a previous empirical study. This allows us to draw more fundamental conclusions about the role of social influence: In particular, we show that the question of whether social influence has a positive or negative net effect on the wisdom of crowds is ill-defined. Instead, it is the starting configuration of the population, in terms of its diversity and accuracy, that directly determines how beneficial social influence actually is. The article further examines the scenarios under which social influence promotes or impairs the wisdom of crowds.