Roland Rudd, the chair of Tate, is in a bullish mood when we meet at his offices in the Adelphi Building, which sits on the Thames between the art institution’s two London sites. “Things have never been better,” he says.

It’s a rebuff to any suggestion that the organisation is in flux – and, as if he were expecting the question to arise, Rudd produces a piece of paper from his suit pocket with notes to prove his point. The recent wins, he says, are so numerous that he has written them down so as not to forget any.

At Tate Britain, Turner and Constable drew in 270,000 people, which Rudd insists “is phenomenal”; Lee Miller was “the most popular photography show anywhere in the UK”; and “Tracey” (Tracey Emin, to you and me) has brought in 125,000 paying visitors, “a remarkable number”, over at Tate Modern.

He’s not finished. “[Visitor numbers] for the end of March were at 6.2 million, about 200,000 up on the previous year.” And don’t forget the membership, which 155,000 people have signed up for (“the biggest membership of any cultural institution … anywhere”).

This picture, as he paints it, is undoubtedly sunny. But, like one of Turner’s maritime scenes, there is a hint of cloud hovering over the flagship. We’re talking less than two weeks after Maria Balshaw stepped down as director of Tate, a position she had held for nine years. The obvious question after Rudd’s laundry list of achievements is, if things are so good, then why is the woman in charge leaving? Rudd says Balshaw is on record saying 10 years would be her limit; by that measure, she isn’t far off. But her successor faces an onerous task.

Part of the reason for the Rudd rundown is because Tate needs some positivity. Visitor numbers have indeed recovered after falling from their peak in 2019, but finances were hit hard during the pandemic. Those financial headwinds have led to multiple rounds of redundancies, restructures and several “culture war” battles that, according to one senior staffer, have left staff morale “on the floor”.

Rudd, then, needs to spin: he’s on the hunt for a new director and wants to sell the Tate as somewhere on the up, rather than the unmanageable, unwieldy “beast” in the midst of an “existential crisis” as some characterise it.

As far as Balshaw is concerned, there is no mystery behind her departure. “You go when things are good,” she says. “You don’t go when they’re bad, and there were some hard years.” Post Tate, she’s been enjoying walking her dog, tending her garden in Kent and making the occasional appearance at openings, such as the new V&A East. Close to a decade is a “healthy” tenure, she says.

Balshaw arrived from the Whitworth in 2017. In Manchester, she had established herself as a driven, instinctive leader who had become a de facto “cultural queen” of Manchester as it underwent an economic transformation in the 2000s. She was praised as a leader with “charm, guts and skill” and was utterly unlike anyone who had ever run Tate.

The first woman to lead the organisation, she went to state school, grew up in the Midlands and had never worked for Tate. In contrast, Frances Morris, the director of Tate Modern when Balshaw arrived, had been with the organisation since 1987; Nicholas Serota, the man she was taking over from, had been in post for 28 years. Alex Farquharson, the Tate Britain boss, hadn’t worked at Tate, but he had known Serota and his wife, Teresa Gleadowe, who taught him at the Royal College of Art in London.

In Tate terms, Balshaw was an outsider. She was viewed as neither an art historian nor a curator, which did not sit well with some. “She was a professional leader,” said one former colleague.

Balshaw balks at that charge. “Those people who said she’s not a curator, they meant, ‘Oh, she hasn’t gone to the Courtauld’ and ‘I didn’t know her from some dinner party’. And that’s just not true. I established myself quickly with curators around my academic knowledge and credentials.”

The institution Balshaw inherited from Serota was on a seemingly endless upward trajectory. He’d willed Tate Modern into existence. Money and sponsorship flowed, including a record-breaking deal with Hyundai, the South Korean car manufacturer, to support 11 years of Turbine Hall commissions. Visitor numbers only headed in one direction.

While there was undoubted momentum, the success of Serota’s tenure made Balshaw’s job all the more challenging. As one former Tate curator put it: “It’s a bit like a new manager coming into Manchester United after Sir Alex Ferguson. Maria was in a very difficult position.”

Early successes included Soul of A Nation, which was in train before she arrived, and Steve McQueen’s Year 3 at Tate Britain. In 2019, a record-breaking number of people visited Tate’s four venues (including Tate St Ives and Tate Liverpool). Then, the pandemic hit.

Instead of the 8 million predicted visitors in 2020, there were only 1 million. The pandemic left a £56 million hole in Tate’s finances and led to several rounds of job cuts. Industrial action followed. At one strike, staff said they were having to resort to food banks and that they just wanted their staff canteen reinstated, which it has been.

The Turner prize, once a jewel in Tate’s crown, has struggled to remain relevant; the Blavatnik Building, which was Serota’s legacy project, is mostly empty. Government funding increases remained below inflation throughout Balshaw’s tenure, and she had to deal with nine culture secretaries in as many years – many of whom seemed to hate the arts. Meanwhile, in the Balshaw era, identity politics and historic questions about race at the organisation became unavoidable, leading to a series of mini crises.

There was the “racist” Rex Whistler mural in the Tate Britain restaurant, and Tate paid out a six-figure settlement to artists who had sued the organisation for discrimination. The most serious incident internally was the handling of the Requiem mural, painted by Chris Ofili at Tate Britain, to commemorate Khadija Saye and the other victims of the Grenfell Tower fire.

Staff felt their opinions on the work, which Ofili admitted was supposed to elicit a “gut punch” reaction, were ignored. Several burst into tears when they saw it for the first time. It was subsequently altered – Tate insiders say this was after staff revolted, Balshaw says it was because of feedback from Grenfell groups. Tate says the claims the mural was changed because of the workforce’s concerns are “completely untrue”.

Balshaw held a meeting about the mural where “people shouted at me, they cried, and they told me what we got wrong and how we could do better”. Was it a mistake looking back? “As I said to the group, most of my emotional energy had gone on thinking about Grenfell United and the next-of-kin groups, and on balance, I think that’s still right.” Balshaw admits she hadn’t anticipated how “tender” the staff were still feeling “because of things like the Whistler mural”.

Rudd is more frank in admitting that mistakes have been made, but he highlights the Hogarth exhibition from 2021, which caused a stir with its labelling that included non-curatorial staff – such as the artist Lubaina Himid – giving their speculative view on the imagery. Critics described it as “wokeish drivel”. Rudd tells me it was “too preachy and too in your face; people didn’t react well to that”.

The fact that staff and the board appear to be so diametrically opposed highlights a central contradiction at Tate: its workforce skews young and progressive while its board and funders lean socially conservative. Any leader has to operate somewhere in that sticky middle.

There’s also the dispute with the National Gallery over its decision to change its collection policy so that it can compete with Tate Modern for 20th-century art. “They have chosen a path, which obviously is competitive with us,” says Rudd. “There’s no point denying that, and would we prefer they didn’t? Of course.” Publicly, Balshaw welcomed the move.

Many people I spoke to said Tate’s woes should not be laid at Balshaw’s door. “Maria has stepped away but we still have a board that doesn’t really know how to answer some of the challenges we face,” said one former senior Tate figure. Many feel the balance of the board is wrong, that it’s too commercially minded (although several people praise Jayne-Anne Gadhia for getting control of the finances).

Frances Morris recalled one board meeting when curatorial staff were asked to explain “what art is” by a trustee. “I thought, how can we have intelligent conversations about mission and vision and decolonising the collection or exploding the canon if they don’t even know what art is?”

The other charge aimed at the Balshaw era is a lack of a clear vision. When the director left, Tate said she had brought “greater gender balance and geographical breadth to new acquisitions”, which sounded remarkably similar to what Tate was saying in 2022 when Morris’s departure was announced.

Balshaw “completely disagrees” with that criticism. She highlights the number of Indigenous artists who were shown at Tate during her tenure and how her Tate consistently ensured Black British creatives were showcased as examples of her approach. David A Bailey, who co-curated Life Between Islands, agrees, pointing out that after his show Lubaina Himid, Hurvin Anderson, Hew Locke and Steve McQueen were all programmed. Bailey’s partner, Sonia Boyce, has a 2027 show at Tate Britain.

Others aren’t convinced. “It needed a new story to take it forward five years ago,” says Morris. “And I think the absence of that long-term vision begins to wear away at the energy in an institution and the sense of enterprise.”

Morris quotes the economist Mariana Mazzucato, who believes institutions always need a mission. “Tate Modern wouldn’t have happened without a mission,” she says. “Tate will not thrive going forward without one.”

Put all that together – the opposite instincts of board and staff; a rough economic backdrop; a ministerial carousel; and calls for a new sense of purpose - and the question of who can successfully take the reins becomes a very hard one. Rudd is in charge of choosing a leader from a longlist of six potentials. Most people in the arts are talking about two names seriously: Karin Hindsbo, the current interim director who was formerly in charge of Tate Modern, and most people’s frontrunner, Jessica Morgan. (Alex Farquharson and Nicholas Cullinan, the British Museum boss, who was in the running in 2017, are two other names floating around.)

Hindsbo is seen by some as a safe “stabilising presence” who has in effect been in charge since January as Balshaw concentrated on the Emin exhibition, a widely acclaimed send-off to her tenure. Tate staff have said Hindsbo has quietly impressed internally by focusing on strategy, managing the collection and reviewing pay, which staff have constantly complained is too low. Rudd says she’s “doing a remarkable job”, although neither he nor Hindsbo would confirm she had applied for the position.

Morgan is the person many people think is the right fit. A powerful fundraiser and curator, she once dealt with the raucous YBAs while working at the Groucho Club in her 20s, and crucially knows the Tate from the inside. The question many are asking is why would Morgan want to give up her sleek, minimalist New York loft and plum job at Dia Beacon, which will be far more lucrative than the Tate director role? Morris described Morgan as a “hugely hardworking individual with a very strong sense of what’s right”, adding: “I think she could inspire.”

The final interview will be held in June, with the Department for Culture, Media and Sport and the Tate trustees overseeing the process. The chosen candidate will be signed off by the prime minister. Whoever comes in will be charged with plotting a new course for Tate at a time when finances are being squeezed and the landscape of British arts is about to enter a new era when free access for all could be abolished.

Balshaw refuses to be drawn on a preferred candidate, but she has her own list of requirements. They’ve got to be “dynamic, progressive and future facing”; “they’ve got to have courage”; “they’ve got to keep being international, especially as Britain is becoming so parochial and polarised”. Crucially, she says, they need to understand that “Tate’s mission is larger than just the UK”. So not much then.