Support for the Prime Minister continued to drain away on Wednesday as he was challenged by some of his top Cabinet colleagues over his handling of the Peter Mandelson affair.

It came as the first Labour MP called publicly for Sir Keir Starmer to resign over the vetting scandal, which has consumed Westminster.

One of Sir Keir's chief allies also refused to back his judgment in making Sir Olly Robbins the fall guy, sacking the Foreign Office mandarin hours after it emerged he had approved Mandelson's security clearance against advice.

And relations with Whitehall deteriorated, with a string of ex–diplomats calling for Sir Olly to be reinstated, while a union boss said he believed Sir Keir was 'losing the ability to work with the Civil Service' as he tried to save his skin at the expense of others.

After another Commons showdown at Prime Minister's Questions, there will be no let–up for the beleaguered PM on Thursday as Cabinet Office Permanent Secretary Cat Little faces questioning by the Foreign Affairs Committee, before which Sir Olly appeared this week.

Next Tuesday, Sir Keir's former chief of staff, Morgan McSweeney, a Mandelson protege said to be instrumental in making him US ambassador, will also appear before the committee.

Mr McSweeney is expected to be asked about the suggestion that – referring to Mandelson's security vetting – he told the Foreign Office: 'Just f***ing approve it.'

He is also expected to be questioned about a mugging last year in which he lost a mobile phone containing potentially revelatory messages with Mandelson.

Within days, the PM could even face a formal Commons investigation into whether he lied to MPs about the saga. Opposition MPs were on Wednesday night talking to Commons Speaker Sir Lindsay Hoyle about a possible vote on establishing a special inquiry into whether the PM told the truth when he claimed 'full due process' had been followed at all times.

The move could lead to an investigation by the Commons Privileges Committee, similar to the probe into claims that Boris Johnson misled MPs over Partygate, which led to the former prime minister being forced out of office.

The potential grounds for an investigation widened on Wednesday when Sir Keir assured the Commons that 'no pressure whatsoever' was applied to officials over Mandelson's appointment.

The claim contradicted evidence from Sir Olly this week that his office and that of the Foreign Secretary were 'under constant pressure' to sign off the appointment.

A Whitehall source said No10 officials were 'panicking' about the unscripted comment, fearing the PM may have to correct it.

The Daily Mail revealed on Wednesday that senior ministers were beginning to publicly distance themselves from Sir Keir.

And it later emerged that some of them had confronted him in person on Tuesday as the scandal dominated their weekly meeting around the Cabinet table in Downing Street.

The PM was warned that his dismissal of Sir Olly – which he again defended in the Commons on Wednesday – risked further damaging relations with Whitehall, making it harder for Labour to enact its policies and killing off any lingering hopes of re–election. Deputy Prime Minister David Lammy said the Government must not create a 'them and us' mentality between ministers and officials, sources told The Guardian.

Three others, including Health Secretary Wes Streeting and Chancellor Rachel Reeves, are said to have urged the PM not to pick fights with civil servants but to keep them 'on side' – an idea Sir Keir is said to have agreed to.

Home Secretary Shabana Mahmood reportedly questioned why Sir Keir had praised Sir Olly as an outstanding civil servant in recent days, having sacked him last week for not telling him that Mandelson had failed vetting. One insider told the Financial Times that Ms Mahmood had said: 'If he's such a great public servant, I don't understand why he has been sacked.'

Government sources insisted that no one spoke out against Sir Olly's sacking, and that the PM had been the first to say that the mandarin's error of judgment did not detract from the work done by civil servants.

But in a fresh blow, one Cabinet minister admitted to having lost confidence in Sir Keir. Speaking anonymously, the senior figure told the i newspaper: 'It's bleak. It's a question for the Cabinet, and colleagues need to come to a view. I know what my view is.'

In broadcast interviews, another Cabinet minister failed to back the PM's judgment.

Pat McFadden repeatedly refused to say whether he thought Sir Olly's sacking was fair.

The Work and Pensions Secretary, a long–standing ally of Sir Keir, would only tell Times Radio: 'As a Cabinet member, I support the Prime Minister's decisions.'

He later added: 'I'm confident the Prime Minister has the strong support of the Cabinet.'

However, a Labour MP became the first to state that Sir Keir should go. Jonathan Brash, MP for Mandelson's former seat of Hartlepool, told GB News: 'I am completely fed up to the back teeth of this psychodrama in Westminster – the own goals... coming from the heart of this Government. It's got to the point where I genuinely think that, as far as the Prime Minister is concerned, it's not a case of if, it's when.'

He added: 'Ultimately, we're in a situation now where I don't think anyone reasonably expects the Prime Minister to lead the party into the next election.'

Another Labour MP said the PM was wrong to sack Sir Olly.

Dan Carden said: 'Reflecting on Olly Robbins' evidence, his sacking for carrying out the political orders of No10 was plain wrong, and its lasting impact will be to further damage the effective running of government that is a responsibility shared between ministers and civil servants.'

Polly Billington, vice–chairman of the Parliamentary Labour Party, refused to be drawn on whether she trusted Sir Keir's judgment. Sir Keir faces the fury of the Whitehall 'blob' for sacking Sir Olly. Lord Butler, who was the country's most senior civil servant for a decade, told the BBC: 'I think the Olly Robbins episode has done a great deal of harm to the relationship between the politicians and the civil service.'

He said Sir Keir had made 'some serious mistakes', but added that a change of prime minister 'wouldn't be in the national interest at this moment'.

Another ex–Cabinet Secretary, Lord Sedwill, told The Times: 'The Prime Minister should retract his accusations against Olly Robbins and reinstate him.'

Dave Penman, general secretary of the FDA union, which represents mandarins, said the mood in the civil service was 'absolutely devastating' and senior officials feared being thrown under the bus 'if it's politically expedient'.

While Nigel Farage vowed to finish off Sir Keir's premiership by inflicting a heavy defeat in the local elections next month.

The Reform leader said the Mandelson 'farce' would force the PM to resign by the end of May.

He told a rally in Barnsley that Labour MPs would desert their leader, adding: 'If you vote Reform, we will get Starmer out, because they'll ditch him.'