"We promised people we weren't going to do this."

There is exasperation in the voice of a long time Labour adviser. But as every hour passes, it is more likely the UK will soon have its seventh prime minister in 10 years.

Talk of Sir Keir Starmer fighting is fading, his exit seems more likely as the weekend goes on. The prime minister is at his country retreat, Chequers, spending time with his wife.

The man coming for his job, Andy Burnham, is spending the weekend with his family, away from home too.

The reasons for Labour to switch leader are compelling. Andy Burnham looks like a winner. He has shown he can beat Reform, who until this moment have seemed a deadly threat to Labour. He is popular in the country, compared to most politicians at least. There are swathes of MPs eager to back him and his brand, believing he's the one who can improve the party's grim position.

"He's an instinctive guy – that's his great talent," said one source.

He's been successful and highly visible as the Mayor of Greater Manchester, known just as Andy everywhere he goes, one of his backers tells me.

He's no stranger to government either, having served as health secretary, culture secretary, and as a Treasury minister years ago. And most of all, Burnham's shown in the Makerfield by-election campaign he has that valuable talent in politics – a capacity to make people feel good.

Labour in Westminster has forgotten what that's like.

There have been more than a dozen big U-turns. Resignations. The mess over Lord Mandelson's job. And after dreadful election results in 2025 and 2026, wipe-out in Wales. Starmer has seemed like a loser to many in his own party.

It is not even two years since his massive win at the general election. But the political perception that he has an appeal to voters? Brutally, that's long gone.

On Friday, the prime minister was still arguing to the cameras that he would fight if Burnham challenges him, refusing to acknowledge that is not an "if", it's a "when".

Even privately some of his backers were still adamant he would run, talking of donors who've given money to run a campaign and office spaces being found.

One source claimed his conversations with cabinet ministers in the afternoon were not about whether he had the authority to stay in office, but the arguments he'd make in a leadership race.

Several sources told me Starmer really does believe he could beat Burnham in a leadership contest, and concluded that a fortnight ago after watching him on BBC Question Time on a Thursday, then failing to explain the borrowing and spending rules in a Newsnight interview on the Friday.

A government insider said: "On Saturday he phoned his closest allies and said, 'I'm sure I could win.'"

But the widespread assumption this weekend in the party is that Burnham would beat him hands down, another government source said: "It's nuts" to imagine the PM could come out on top.

An increasing number of ministers, previously loyal to Starmer now think it's time, as one cabinet source told me, they "wouldn't want the prime minister to humiliate himself" in a race.

The chances of him staying to fight are diminishing. But what is still a mystery this weekend is exactly how Starmer will respond.

One of his colleagues' frustrations with him has always been that he seems unknowable.

"It's very hard for people to know a person who doesn't know themselves," said another government insider.

Don't underestimate the anger towards Burnham inside Downing Street, and that's shared by some other ministers too.

Not just for what's happening now, but how they see he's chipped in unhelpfully from the sidelines since the day Starmer moved into No 10.

One Starmer ally told me: "This is not a chase, these are big decisions about who is going to run the county – it can't be rushed 20 minutes after a by-election."

Many in Labour aren't sure what Burnham would actually do in office either.

The former minister, Jess Philips, told the BBC this morning that Burnham or any other candidates must be "tested with the rigour of at least some manner of contest".

One government minister, Mike Tapp, told me bluntly he'd never met him, and "I don't know his politics".

A backer of the prime minister said, when Burnham had faced tough questions in the by-election campaign, "he's fallen apart".

There's also concern about the precedent of ousting a leader off the back of a by-election, the votes from a group of only 77,000 people deciding everything for the whole country. Burnham would have no mandate from the public, without a general election.

And what happens if Labour's standing didn't improve? Might those calling for a removal van for the current prime minister do the same again? What if there were another by-election when Prime Minister Burnham was in trouble?

Is it mad to imagine that other big names from the past - David Miliband or, even Ed Balls - might abandon New York and the breakfast TV sofa, and fancy a comeback too?

Just as there are compelling reasons for Labour to make the switch, there are serious risks. There may yet be a contest, and another candidate aside from Wes Streeting could find the 81 names to run.

But with 100 MPs now calling for Starmer to go and support for him to stay in the cabinet fading, one senior party figure predicts "he'll realise this weekend that he can't keep the Cabinet and ministers together and will have to go".

Labour has found itself in a strange situation it promised you it would never reach – en route to removing it's first prime minister to win, in 14 years. And congratulating themselves for winning a seat they already held, so they can get rid of the man whose campaigning won them all the seats they have.

But whether Starmer reaches the conclusion that he will have to go himself, or he is forced into a contest, more agree with another veteran figure, "It's done".

Starmer's success in getting Labour back to power after the crash of 2019 was remarkable. But his time in office has proved a very different story, of many frustrations and failure.

One party source told me: "My experience of working with his administration is - the fundamental part of job of prime minister and Labour Party leader is to be a political leader and he is neither political nor a leader.'"

That is brutal. But many in Labour would agree it's true, even though they'd point to achievements during his time in power – progress on pushing down NHS waiting lists, immigration coming down, his handling of foreign affairs and a growing economy.

The vow not to repeat the Conservatives' habit of switching prime minister might be the last political promise Starmer breaks.

BBC InDepth is the home on the website and app for the best analysis, with fresh perspectives that challenge assumptions and deep reporting on the biggest issues of the day. Emma Barnett and John Simpson bring their pick of the most thought-provoking deep reads and analysis, every Saturday. Sign up for the newsletter here