The top civil servant in the Foreign Office has been forced out after a Guardian investigation revealed his department overruled a decision to deny Peter Mandelson security vetting clearance.

The departure of Sir Olly Robbins follows the disclosure, based on multiple sources, that Mandelson had in January 2025 failed a developed vetting process, a highly confidential background check by security officials.

Keir Starmer had by then announced he would be making Mandelson the UK’s chief diplomat in Washington, posing a dilemma for officials at the Foreign Office, who decided to use a rarely used authority to override the recommendation from security officials.

News of Robbins’ departure came after Downing Street spent much of Thursday battling demands for the prime minister himself to resign over the affair. No 10 insisted that the prime minister had no knowledge that security officials advised Mandelson should not be given clearance, and said responsibility lay with the Foreign, Commonwealth and Development Office (FCDO).

In September, in a letter co-signed with foreign secretary Yvette Cooper, Robbins told MPs that “conducted to the usual standard set for developed vetting.

The Guardian understands that Robbins was told he had to resign after Starmer and Cooper lost confidence in the civil servant.

Mandelson’s failure to secure vetting approval has not previously been publicly revealed, despite intense scrutiny over his appointment and the release by the government of 147 pages of documents supposed to shed light on the case.

Before news of Robbins’ departure, a government spokesperson said: “The decision to grant developed vetting to Peter Mandelson against the recommendation of UK Security Vetting [UKSV] was taken by officials in the FCDO.”

“Neither the prime minister, nor any government minister, was aware that Peter Mandelson was granted developed vetting against the advice of UK Security Vetting until earlier this week.

“Once the prime minister was informed he immediately instructed officials to establish the facts about why the developed vetting was granted, in order to enact plans to update the House of Commons.”

Downing Street insiders said he was “furious” about having been kept in the dark about the vetting decision.

Earlier on Thursday, the Conservative leader, Kemi Badenoch, said that if the prime minister had misled the House of Commons over Mandelson’s vetting he “must take responsibility”. Ed Davey, the leader of the Liberal Democrats, said: “If Keir Starmer has misled parliament and lied to the British people, he has to go.”

Further documents are due to be released. However, it can also be revealed that senior government officials have been considering whether to withhold from parliament documents that would reveal Mandelson was not given vetting approval from security officials.

The decision, which rests with the Cabinet Office, has not yet been taken. Any attempt to withhold the documents from the intelligence and security committee (ISC) could amount to a breach of a parliamentary motion to release “all papers relating to Mandelson’s appointment”.

In the statement released on Thursday evening, the Downing Street spokesperson said the government was “committed to complying” with the parliamentary motion, called a humble address. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.”

The revelation that the now former ambassador was not granted clearance by UK UKSV, a division of the Cabinet Office that scrutinises the background of prospective civil servants, will raise further questions about the prime minister’s judgment in appointing him.

Starmer will also be pressed over whether he misled the public in remarks about the security vetting process, which he said had given Mandelson “clearance for the role”.

It is not known who in the Foreign Office made the decision to overrule UKSV.

Robbins was the department’s top civil servant in late January 2025 when the decision was made, having taken up the role earlier that month. The foreign secretary was David Lammy, who is now the deputy prime minister.

Starmer’s then chief of staff, Morgan McSweeney, who resigned in February over his role in appointing Mandelson, could also now be asked whether he had any involvement in, or knowledge or, the decision to overrule UKSV’s denial of clearance.

Friends of McSweeney told the Guardian that he had no knowledge of Mandelson’s developed vetting process or the outcome.

That decision was made weeks before Mandelson was due to take up his post in February 2025. Seven months later, he was sacked over his relationship with the convicted sex offender Jeffrey Epstein.

Ministers and officials are now likely to be pressed over whether they have been fully transparent about the process that led to his appointment.

At a press conference in Hastings on 5 February, Starmer responded to a question from a journalist by saying there had been “security vetting, carried out independently by the security services, which is an intensive exercise that gave him [Mandelson] clearance for the role. You have to go through that before you take up the post.” He added: “Clearly both the due diligence and the security vetting need to be looked at again.”

This appeared to partly put the blame for Mandelson’s appointment on the failure of a vetting process which, according to sources, his government had overruled.

As a result of Mandelson’s sacking as US ambassador on 11 September 2025, after the extent of his relationship with Epstein came to light, parliamentary scrutiny mounted. On 16 September, Yvette Cooper, the foreign secretary, and Robbins, then her top official, responded to questions over the vetting process with a letter to the foreign affairs select committee.

“Peter Mandelson’s security vetting was conducted to the usual standard set for developed vetting in line with established Cabinet Office policy,” the letter said, explaining that the process had been undertaken by UKSV on behalf of the FCDO.

Cooper and Robbins said the process had “concluded with DV clearance being granted by the FCDO in advance of Lord Mandelson taking up post in February”.

What the letter failed to inform parliament was that UKSV had denied Mandelson’s clearance – a recommendation that threatened Starmer having to withdraw a high-profile appointment he had already made public.

UKSV’s vetting decisions are almost always enforced by government departments, but they technically have the authority to override the recommendations. The precise reason that UKSV recommended that Mandelson not receive clearance is now likely to be subject to intense speculation.

According to publicly available government documents, the UKSV developed vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history.

It is understood that developed vetting, which is a requirement even for junior civil servants in the Foreign Office, almost always ends in approval, although there is sometimes a recommendation to manage risks. In some cases, approval is subject to mitigations. An outright denial, as in Mandelson’s case, is rare.

It is even more exceptional for a decision by UKSV to deny an applicant to then be overruled by a government department, although officials do have that power. The decision to do so in the case of Mandelson occurred over a 48-hour period in late January 2025.

A formal decision to deny him clearance was made by UKSV on 28 January 2025. According to sources, UKSV informed the Foreign Office that the risk factors involving Mandelson meant that his clearance should be denied.

Robbins was a few weeks into the role of permanent secretary at the time of the decision, and was Mandelson’s soon-to-be line manager, both of which suggest he would have been involved in the decision. While a decision of that magnitude raises questions about whether there was political approval, or consultation, sources close to Lammy insisted that he only became aware of the vetting override on Thursday.

By 30 January, Mandelson was told by the Foreign Office that his security clearance had been “confirmed”. It is not known whether he was ever informed that UKSV had not approved his application.

In November 2025, after Mandelson’s resignation, MPs on the foreign affairs select committee questioned Robbins at a hearing about the former ambassador’s vetting process.

Robbins said the “vast majority” of vetting reports were “relatively straightforward”. He added: “Ones that require more senior judgment, and potentially a discussion about managing and mitigating risks, are escalated appropriately.”

Asked directly if Mandelson’s appointment had been “escalated”, Robbins replied: “I certainly cannot comment on that.”

Starmer’s decision to appoint Mandelson – the first political appointee to the role, as opposed to a career diplomat, since 1977 – despite well-documented concerns about his suitability, has prompted the most serious crisis of his premiership.

When McSweeney resigned in February, he said that he took “full responsibility” for advising Starmer to appoint Mandelson, and implied there had been shortcomings in the vetting process. “While I did not oversee the due diligence and vetting process, I believe that process must now be fundamentally overhauled,” he said.

Mandelson’s vetting by UKSV was one of two separate processes that involved scrutinising his background.

The first, which occurred because he was a political appointee, was a due diligence process that occurred before the announcement of his role. It involved the propriety and ethics team (PET), a group of officials in the Cabinet Office.

This was largely based on publicly available material, with a report produced by PET for Starmer, summarising Mandelson’s professional and financial relationships, reputational risks – which included his relationship with Epstein – and his previous roles. The prime minister ignored those warnings and went ahead with the appointment.

Far less attention has been given to the second, entirely separate process undertaken by UKSV in the weeks after his appointment was announced but before he took up the role.

The Foreign Office, the Cabinet Office, and Peter Mandelson have been approached for comment.