Labour has spent much of the past year paralysed by competing fears. MPs’ dread of facing voters with Keir Starmer as prime minister has been kept in check by their recoil from the process of replacing him. They know the prime minister is an electoral liability; they know that the electorate takes a dim view of chaotic, regicidal parties that showcase disunity and factional rancour when they are supposed to be running the country.

Impatience with Starmer’s leadership has, until now, been neutralised by reluctance to gamble on a contest that might replace him with someone worse. Last week’s local and devolved ballots changed the calculus. Labour MPs now have indisputable evidence that they are cruising towards nationwide electoral oblivion. A growing number think the trajectory will not change if the leader stays the same.

The results were catastrophic by any measure, but that wasn’t the only factor provoking backbench demands for Starmer’s departure, or the flurry of frontbench resignations. The prime minister’s response exemplified traits that colleagues find infuriating about his leadership. He took responsibility for Labour’s electoral evisceration in terms that were more defiant than humble.

In an interview over the weekend, Starmer said he intended to serve a decade in Downing Street. In a speech on Monday, Starmer characterised voters’ damning verdict on the two years of Labour government as the steep part of a normal learning curve for new prime ministers. The remedy for public frustration was not a different direction but the current one pursued with greater urgency. He said that “incremental change won’t cut it”, while proving with caveat-laden half-pledges that increments are the only currency he holds.

The deficiency was starkest on the subject of Brexit. The prime minister was scathing of its consequences. He denounced Nigel Farage’s evasion of accountability for a litany of broken Eurosceptic promises. He pledged a return to “the heart of Europe”. These are things he may have believed in opposition but failed to say in the election campaign that brought him to power. Now, in what was advertised as a moment of rhetorical unshackling, he still couldn’t commit to break free from the red lines – no single market membership; no customs union – that confine Britain to the economic periphery of its home continent.

He said the process of trying to restore hope and security to the country since winning power had revealed the need for “a bigger response than we anticipated in 2024”. The plural pronoun is revealing. Who are “we” in that analysis? Labour MPs never doubted the scale of the challenge. Plenty thought the manifesto on which they stood was too timid, but they were cajoled or bullied into accepting modest ambition as the price for reassuring voters that Starmer had neutralised his party’s radical impulses.

It seemed like a good deal, considering the record of failure by Labour opposition leaders to reach Downing Street by any other route. A rare landslide victory then proved the point.

If there ever was a definition of Starmerism – and the prime minister has always insisted no such word exists – it consists in that calculation. It was the view that Britain had been brought low by incompetent, dogmatic rightwing government, ineffectively countered by unrealistic, fanatical leftwing opposition. If polarised ideology was the problem, the solution must be centre-hugging pragmatism.

The change that voters craved might thus be embodied in a dull but worthy prime minister who would apply himself to the business of government with meticulous focus on problem-solving. Starmer’s few remaining defenders say those qualities are the right ones and tragically undervalued in an age of public contempt for politics sustained at a pitch of relentless fury.

Generous critics concede that Starmer is a scrupulous public servant, but note that a diligent pragmatist should have developed a fuller programme for government when still in opposition. It was naive, at best, to assume that the mere act of replacing wicked Conservative ministers with noble Labour ones would unblock the sluices that had apparently prevented good policy flowing out of Whitehall.

The harsher judgment is that the Starmer project made a fetish of pragmatism as an electoral tactic to the exclusion of policy; that avoidance of awkward questions – how to raise money for public services, how to repair the damage inflicted by Brexit – amounted to a ban on thinking about answers; that the determination to purge Labour of Jeremy Corbyn’s legacy was pursued with factional monomania that mislabelled dissent of any kind as toxic leftism.

The vast majority of MPs desperately wanted to support their leader. But they have struggled to discern what they are being loyal to when the government’s most familiar manoeuvre is the U-turn, its fiscal mandate was set to parameters chosen by the last Conservative government and its immigration policy sounds like a queasy tribute to Farage.

If ministers can’t confidently articulate the point of their government, it isn’t surprising that voters look elsewhere for clarity of purpose and validation of their grievances.

Starmer’s drab oratory doesn’t help. But poor communication is most often a symptom of ill-defined policy and uncertain purpose. He could have been a more compelling messenger if he had known what message he wanted to send.

In opposition, it was the single word “change”. That was easy to initiate but hard to substantiate. Any credit available to the incoming prime minister by virtue of his not being a Tory expired on the threshold of No 10. From day one, most of Fleet Street treated the Labour government not as a legitimate manifestation of democratic preference, but as the accidental side-effect of voters’ haste to be rid of the Conservatives.

Every scandal and mistake degraded the sense of difference from the old regime. Starmer had been an empty receptacle in which voters placed their hopes of renewal. Without momentum in any direction, he then became the repository for all the cumulative resentment of politicians who promise everything and deliver nothing.

This helps explain the intensity of anger that Labour canvassers encountered towards their leader on the campaign trail – a venomous loathing that shocks even deeply disillusioned MPs. It bears no resemblance to the gentler critique of Starmer as an honourable statesman whose administrative capabilities have been squandered from lack of a coherent creed.

For a party staring at possible annihilation, it doesn’t really matter whether voter contempt for its leader is unfair if it is also irreparable. Starmer’s refusal to accept that he is the problem, prescribing more of himself as the solution, is a major factor turning private misgivings into public demands for new leadership. The last reserves of goodwill have been depleted by the feeling that the prime minister is rather too attached to his own self-image as a man of principle. What he presents as civic duty to continue serving the country looks more like refuge in arrogant denial.

Many of his predecessors have ended up in the same place. The sheer intensity of the job cultivates a particular conceit in those who do it, believing none of their colleagues could possibly be equal to the task. Often they are right. Recent British politics has plenty of case studies in how not to succeed as prime minister.

The candidates now manoeuvring in half-shadow to be the next Labour leader must believe that it will be different for them. That confidence is a psychological function of the ambition that propels people to the top. Starmer cultivated it as leader of the opposition, watching three Conservative prime ministers fail. He thought he could be the change the country craved. It wasn’t enough. Not even close. So what was missing? When did it go wrong?

Removing Keir Starmer is a remedy to the condition of having Keir Starmer as leader. Nothing else. It isn’t a diagnosis of what the country has been lacking or a destination it should be reaching. Anyone imagining they could replace the incumbent should have the confidence to express those things now. Make the case for a contest by displaying a credible alternative. Otherwise, the only prize of succession is to become the new face of the same old problem.

• The future starts with us: Gordon Brown in conversation.
  On Thursday 10 September, join Hugh Muir and Gordon Brown to discuss the intricate connections between global instability and civic decline, as explored in Brown’s new book, The Future Starts With Us. Book tickets here or at guardian.live

• Rafael Behr is a Guardian columnist