The Whitehall satire Yes Minister was said to be Margaret Thatcher’s favourite TV show due to its proximity to reality, as the programme’s loquacious top civil servant, Sir Humphrey, might have put it.

Yes Minister had a familiar groove: there would be a problem in response to which the mandarin would artfully deploy the most astonishing sophistry to avoid blame or get his own way. Jim Hacker, the largely clueless yet ambitious politician played by the late Paul Eddington, rarely won the day.

This week, in a most public way, Keir Starmer, had his own Sir Humphrey moment – potentially an existential one.

Olly Robbins, fired by the prime minister as the Foreign Office’s permanent secretary after not informing him that Peter Mandelson had failed his vetting for the role of ambassador in Washington, appeared before the foreign affairs select committee, as many a civil servant embroiled in a crisis has in the past.

How was it that Mandelson took up the job, given that vetting result? “I was told – let me be completely precise – that UKSV [UK Security Vetting] were leaning towards recommending against, but accepted that it was a borderline case.”

Was it not a disgrace that the PM was not informed? “The government makes very clear in public guidance that for anybody to be briefed, outside the vetting security process, on the information into or findings of a UKSV process has to be for, I think I quote, ‘wholly exceptional circumstances’.”

Cabinet ministers, while understandably scratching their heads, have emerged from it all questioning the PM’s judgment.

Watching on from the sidelines was the former Conservative prisons minister Ann Widdecombe. She said she had seen this movie before: the politician rarely emerged as the hero.

On 13 May 1997, the veteran newshound Jeremy Paxman asked Michael Howard, the former home secretary, a question 12 times on the BBC’s Newsnight: “Did you threaten to overrule him?”

The “him” was Derek Lewis, the former director general of HM Prison Service, and Howard was being accused of having gone beyond his powers by ordering the mandarin to fire a prison governor. It was part of a wider dispute between Widdecombe and Howard over whether he had misled the Commons about the reason behind Lewis’s own subsequent sacking.

Widdecombe accused Howard, who was seeking election as leader of the Conservatives after their 1997 trouncing at the hands of Tony Blair, of having “something of the night” about him.

In turn, one of Howard’s supporters claimed to the Daily Mail that Lewis had “wooed” Widdecombe with flowers and chocolates. “He hadn’t sent me a petal – and because of my girth no friend would buy me flowers,” said Widdecombe.

It was not just in the complexity of the story that Widdecombe said she saw parallels with the recent row – but in the inevitable damage inflicted on those turning on highly effective civil servants, as both Lewis and Robbins are widely regarded as having been.

“Firstly, they know where the bodies are buried,” said Widdecombe of such senior civil servants. “Secondly the public do not like scapegoating, that always puts them off. Thirdly, it opens the person who has done the sacking to all sorts of scrutiny and criticism, which is what happened to myself and Michael.”

Lewis received a six-figure payout after taking the government to court. Widdecombe noted the general secretary of the civil servants union was sitting behind Robbins on Tuesday. “The union took up Derek’s case too,” she said.

Similar episodes spilled out during the Blair administration.

After falling out with the transport secretary Stephen Byers in 2001, Martin Sixsmith received a £250,000 payout (worth about £500,000 today) after his resignation as director of communications was announced when he had not actually resigned.

But perhaps the most infamous clash between mandarin and the political class involved David Kelly, the government scientist whose identity as the likely source of a BBC report about a “sexed up” intelligence report on Iraq’s alleged weapons of mass destruction, was confirmed by the Ministry of Defence under the order of the then defence secretary Geoff Hoon.

Kelly subsequently endured a horrendous time in front of the foreign affairs select committee on a hot day.

Donald Anderson, now a Labour peer but then an MP chairing the committee, recalled that Andrew MacKinlay, then the Labour MP for Thurrock, accused Kelly of being “chaff” and a “fall guy” and berating him for not being frank about which journalists he had spoken too. Kelly killed himself two days later.

Anderson explained that MacKinlay had “chaff” on the mind as he had been on a recent trip to Iraq where it had been used to avoid being shot down. “It wasn’t meant to demean David Kelly, but he got into a great deal of trouble with many death threats as a result,” Anderson said of MacKinlay.

As well as a human tragedy, Kelly’s death was a disaster for the government, with Blair’s position in Downing Street seemingly under threat for a while.

Is it always the case, then, that the politicians end up in trouble when they go up against the civil service?

“Not necessarily,” said Anderson. “Robbins gave a stout defence. I’m not sure what the protocols are, but it does seem, as the prime minister said in parliament, ‘incredible’ that Robbins did not inform him.”

Ivan Rogers, the UK’s permanent representative in Brussels until his resignation in January 2017, suggests that if you step back, the prime minister’s appetite for throwing Robbins off a cliff should worry anyone who believes in an impartial civil service.

Rogers quit Theresa May’s government after a note he had written warning of a long transition period before the conclusion of the Brexit process was leaked to the BBC.

The leak fuelled the claim that Rogers and others in the deep state were “secret remainers” – a perception that some in Downing Street appeared happy to promote as they sought to keep May, who had voted remain, onside with the Brexiters.

Rogers suggested that politicians’ habit of attacking the civil service in such a public way was only bolstering the view of some that only political-aligned officials could be relied upon to get things done. “I think it’s a bit of a systemic crisis,” he said and suggested there had been a collapse of faith in the values that underpin the civil service: “impartiality, integrity, you serve masters and mistresses of both parties or all parties, you just get on with it”.

The trend started, Rogers suggested, with Blair wanting “true believers” around him in Downing Street, as opposed to those sympathetic to Gordon Brown. Brexit put a bunsen burner under that politicisation, with ministers now often looking for policy to be formulated outside departments rather than rely on internal expertise, he added.

Philip Rutnam, the former permanent secretary in the Home Office, was another who arguably fell foul of the scepticism among the political class about the impartial civil service.

Rutnam resigned in 2020 with the announcement he was going to sue the government for constructive dismissal after becoming the “target of a vicious and orchestrated campaign” led by the aides of the then home secretary Priti Patel.

Rutnam had raised his concerns about Patel’s bullying behaviour earlier in the year which had leaked. He was then made the focus of attacks in the press. “It was all playing day after day in the media,” he said. “I was unable to respond to the untrue things being said because any statement I made had to be approved by the same special advisers.

“Finally I decided to resign and to take the government to an employment tribunal. It was only when I had resigned that I could speak freely again, and of course there was huge media coverage.”

For Rutnam, one of the key dynamics today is that the speed of the media cycle has made it difficult for politicians to bide their time and act with due caution.

“I think Starmer made a mistake in summarily dismissing Sir Olly last week,” he said. “He could have commissioned an urgent investigation into why the Foreign Office had not informed him, [the then cabinet secretary] Chris Wormald or others about the vetting process …

“I became a civil servant in 1987 just after Margaret Thatcher had won her third general election. If I look at the arc of time between then and when I left six years ago, the biggest single change is the intensification of the media cycle. That has really affected politicians’ behaviour and driven the rise of special advisers. I don’t think our constitutional system has adapted sufficiently.”

Rutnam added: “All of this mess could have been avoided in the Robbins case if only the original concern had been properly handled by No 10. Instead there was spiral after spiral – good for the media, but bad for everyone else involved.”