Keir Starmer’s grip on power appeared to be slipping away on Monday as more than 70 Labour MPs publicly urged him to set out a timetable for his departure, despite his pledging to fight any challenge.

The prime minister warned the country would “never forgive” Labour for plunging into the chaos of a leadership election – and that he intended to prove his doubters inside and outside the party wrong.

One cabinet source suggested that if the flood of MPs calling for him to go continued, two or three cabinet ministers could take the prime minister aside to advise him to consider the political reality. They stressed that they would not try to force him to quit.

But Starmer looked increasingly isolated as momentum built among previously loyal MPs from across the party calling for him to step down – while his cabinet ministers mostly stayed silent.

Among them was Wes Streeting, whose close allies were among those urging Starmer to set out a “swift” timetable, in what appeared to be an orchestrated move. But one close friend suggested the health secretary had “no plans to pull the house down”.

Supporters of Andy Burnham also published letters calling for the prime minister to go. Sources on Labour’s ruling national executive committee, which blocked the Greater Manchester mayor from standing in the Gorton byelection, suggested they could take a different course next time.

“The officers group could move if there was clearly a question mark over political authority. But there would have to be a byelection called first before we know whether that is the case,” one said. Another said “things could move” should Starmer clearly not have support to continue as PM.

Angela Rayner, whose own chances of running are stalled until her tax affairs are resolved, appeared to throw her weight behind Burnham, arguing that blocking him had been a mistake and he should be allowed to return to parliament.

Her stock as a potential successor to the Labour crown has fallen in recent weeks, as Burnham’s has risen, with allies suggesting she was no longer determined to go for the top job herself, and could instead be part of a team. But she would still be expected to run if Burnham could not.

Starmer and his allies struggled to fight a rearguard action to shore up his leadership, unlike after Anas Sarwar, Labour’s leader in Scotland called for him to go in February, when cabinet ministers came out en masse in support.

Starmer survived an immediate threat after Catherine West, a stalking horse candidate, changed course and said she was gathering names of MPs who supported an orderly transition instead.

More than 70, including several junior backbenchers, from across the party’s ranks said Starmer had failed to convince them he had what it took to lead the country into the next election. They represent 25% of the party’s backbenchers.

In a speech earlier on Monday, Starmer said he would not resign and would prove his doubters wrong, and he would fight any leadership challenge.

“I take responsibility for not walking away, not plunging our country into chaos, as the Tories did time and again, chaos that did lasting damage to this country. A Labour government would never be forgiven for inflicting that on our country again,” he said.

“I know that people are frustrated by the state of Britain, frustrated by politics, and some people are frustrated with me. I know I have my doubters, and I know I need to prove them wrong, and I will.”

One cabinet ally of Starmer urged MPs to pause. “Everybody needs to calm down and take a deep breath. If you want to be seen as a credible governing party, the idea that you come into power promising stability, and then 20 months later decapitate your leader, is just madness,” they said.

But Starmer’s speech did not stem the steady flow of statements from MPs which called for the prime minister to announce an orderly transition.

At Monday lunchtime, the co-chair of the Labour Growth Group, Chris Curtis, once considered a loyalist group, said Starmer could not deliver the change the country needed. “I therefore think it’s time for us to look for new leadership,” he said.

“And I think what that means is the prime minister rightly now setting out a timetable and an orderly process for a leadership election, and one in which Labour has a discussion about the vision for the country and what changes we think are needed in order to face the very real challenges that Britain currently faces.”

Curtis was the first MP known to be close to the health secretary, Wes Streeting, who called for Starmer to go. Other MPs have followed, including the Scottish Labour MP Alan Gemmell and Streeting’s constituency neighbour Jas Athwal, as well as his parliamentary private secretary, Joe Morris.

Allies of Streeting said that he had no plans to speak on Monday and reiterated his support for the prime minister. However, some MPs believe he is on the brink of declaring he will run. Others suggested a big beast on the soft left – like Ed Miliband – would run just to stop him.

MPs often cited the effects of the prime minister’s personal unpopularity on the local election result. Lorraine Beavers, the MP for Blackpool North and Fleetwood, said: “Without a massive change in approach, when elections are held in my corner of Lancashire next year, voters will send the same message with even greater force.”

The Birmingham Erdington MP, Paulette Hamilton, said: “On the doorstep, voters repeatedly told us the same thing, national issues and the party leadership meant they could no longer vote Labour.”

Later in the day, three other junior frontbenchers – all PPSs – also called for Starmer to go. Sally Jameson from the Home Office, Tom Rutland from the Department for Environment and Melanie Ward, David Lammy’s aide at the Ministry of Justice.

Others who have called for the PM to go include many of the 2024 intake, including David Smith, Luke Myer, Markus Campbell-Savours, Tony Vaughan, Fred Thomas and Sarah Smith. Others included ex-ministers Catherine McKinnell and Justin Madders.

But others put out statements saying they feared a change of prime minister would cause chaos at a time of economic turmoil and play into the hands of Nigel Farage. The Welwyn Hatfield MP Andrew Lewin said on his Substack: “I have learnt enough to identify a trap and not to walk straight into it.

“If we grant Farage his wish today, it would not only be a win for our opponents, it would do lasting damage to both the country and the Labour party’s reputation for being able to govern.”

In a Labour MP WhatsApp group, Natasha Irons, MP for Croydon East, wrote: “Bottom line, changing leader because Nigel Farage has forced us to is not something any of us can come back from. Anyone who thinks we can needs to wake up.”

Another MP said: “I think we have to face up to the fact everyone one of them is fucking useless. Andy’s strategy has been a disaster. Angela has bottled it. Ed is clearly hiding to nothing. Wes is AWOL. God knows what Catherine West is doing. I am not quite sure how we ended up here.”