The most difficult thing in life – any life – is to admit you are wrong. What’s true for someone in the privacy of their own home is massively more true for the prime minister. But last Thursday, Keir Starmer made a mistake. That afternoon, the Guardian reported that Peter Mandelson had failed his security vetting and the permanent undersecretary at the Foreign Office had overruled the recommendation of UK Security Vetting (UKSV), a department within the Cabinet Office. Within the news cycle, the prime minister and foreign secretary announced they had lost confidence in Olly Robbins and he was forced to resign.

Over the past few days, No 10 has doubled down on its version of events. On Monday, Starmer told the House of Commons that, if he had known Mandelson had “failed” his vetting, he would not have allowed his posting to Washington to proceed.

Before dawn at JFK airport yesterday, I watched Robbins give evidence to the foreign affairs select committee on iPlayer while waiting for my flight to Heathrow. Emily Thornberry, the chair of the committee, and her colleagues were polite, forensic and steely. Robbins answered in the same spirit. He set out the facts. He spoke only from his knowledge. He refused to breach the confidentiality of the vetting process and he continued to support colleagues who cannot defend themselves in public, despite the fact that he was no longer a civil servant.

The session made plain that security vetting is an art, not a science. Its sole objective is to protect national security. Rarely does anyone outright “fail” the process – although UKSV did take the view that Mandelson should be “denied” approval because of an overall “high concern”. Nonetheless, decisions are always a matter of judgment about whether vulnerabilities can be mitigated, not a matter of filling in a template.

Three things were clear from the session – and anyone agreeing with those three things should accept an inescapable conclusion.

First, by the time Robbins took over as permanent undersecretary on 20 January 2025, the prime minister’s decision to post Mandelson to Washington was irreversible: the posting had been publicly announced without caveat, the king had approved it, and the White House had given agreement. Officials at No 10 had only reluctantly accepted that security clearance was required at all. Robbins knew that the problems with the posting had already been laid out to Starmer, whose enthusiasm for it was undimmed. Vetting was a formality that had to be delivered.

Second, Robbins said that the Foreign Office’s director of security reported to him at their meeting that UKSV “considered Mandelson a ‘borderline’ case, leaning towards recommending that clearance be denied”. (Such clearance was necessary for developed vetting, or DV – the level needed by officials in roles that require them to have “frequent and uncontrolled” access to top secret material and assets.) The Foreign Office is an intelligent customer that interrogates the agency’s work. There was a discussion about the agency’s findings and a decision that, with mitigations in place, Mandelson could have his DV. As Robbins made clear, nothing was overturned.

Third, the whole process of security vetting is confidential. Even permanent undersecretaries see only what they need to know. Like Robbins, the only completed forms I ever saw as permanent undersecretary were the ones about me. Ministers need to know only the result. It’s similar to medical clearance. Everyone posted overseas needs medical clearance; the chain of command needs to know it’s been granted, they have no proper interest in the detail.

Robbins did his job, aware of the pressure from across Downing Street but not buckling to it. And yet misunderstanding about what that job required led the prime minister to rush to a wrong judgment. I cannot believe that, had he waited until after the foreign affairs select committee session, the PM would have sacked Robbins.

The world is an uncertain place. The Foreign Office and its professional head are dealing with simultaneous crises in Ukraine, the Middle East and the transatlantic relationship. Britain cannot afford a gap at the top, nor can it afford to lose the services of a first-class civil servant whose diligence and thoughtfulness were on full display yesterday in Portcullis House. There is one immediate conclusion in my view: the government should reinstate Robbins as permanent undersecretary.

• Lord McDonald of Salford was permanent secretary at the Foreign Office, 2015-2020, and is now a crossbench peer