The creation of hope is a vital but risky part of democratic politics. Leaders or would-be leaders who arouse hope attract supporters, motivate activists, achieve momentum and win over voters – and then have a chance of holding together political parties, governments and societies in harder times. From Barack Obama to Clement Attlee, Salvador Allende to Zohran Mamdani, leaders from across the left in particular have heavily relied on hope to launch and sustain their ruling projects.

Meanwhile, an absence of hope has quickly doomed other left of centre governments. Keir Starmer’s decision, only eight weeks into his premiership, to summon the media to the Downing Street garden and tell them that “things will get worse before we get better” in the UK was a mistake from which his administration never recovered. In a society where most lives have been getting harder since the 2008 financial crisis, Starmer’s downbeat manner, however justified by the deep problems he inherited from the Tories, was not an emotional register that much of the electorate desired.

Andy Burnham seems less likely to make that error. A more emotionally literate politician, he crammed his Makerfield victory speech with upbeat language. “Tonight could, just could, be the turning point,” he said. “Bringing back something we’ve lost – hope – hope for the future … There is a chance now … to build a new politics based on unity and hope. Turning away from the path that takes us to a divided, dark politics.”

At times, his surge towards Downing Street, despite all the obstacles placed in front of him by the Labour right and Reform UK, has been a seductive advertisement for the power of positive politics. On Monday, as I was reading an expansive set of policy proposals for a Burnham government titled The Productive State, written by two of his more radical backers, Mathew Lawrence and Alex Williams, a Sky News helicopter hovered noisily outside the window, covering Burnham’s almost triumphal arrival in London by train. Some Labour supporters and politicians, such as all the MPs who crowded in for a selfie with him, are allowing themselves to feel slightly optimistic for the first time in two years.

And yet, even more than most parties, Labour has a tricky relationship with hope. Founded by an unstable alliance of idealists and pragmatists, radicals and self-styled realists, it has been afflicted by arguments between them ever since. Broadly speaking, the right of the party sees the left as too hopeful about how much can be done to make Britain more equal, while the left sees the right as not hopeful enough. The current briefing and counter-briefing of journalists by anonymous Labour figures about who should hold the key posts in a Burnham government – disruptive reformers such as Ed Miliband, or cautious adjusters of the status quo such as Pat McFadden – is the latest episode in this 126-year-old argument.

There is more agreement, however, both inside and outside the party, about other aspects of Labour’s ambiguous relationship with hope. It is widely believed that the party needs an optimistic message to perform well at general elections. That message can be leftwing, as under Attlee in 1945, Harold Wilson in 1964 and Jeremy Corbyn in 2017; or centrist, as under Tony Blair in 1997 and 2001, and Keir Starmer in 2024.

Yet, as most of these examples suggest, Labour optimism also often brings problems: creating excessive expectations, underestimating the complexity of changing Britain and downplaying the enemies and barriers the party always faces. For critics of Labour from both left and right, disappointing voters in office is the party’s key characteristic.

A sterner critique of the party still is that the hope it offers is essentially a trick: presenting the softening of Britain’s economic and social hierarchies as instead their fundamental reform. Until recently, that critique of Labour was largely confined to much smaller, more leftwing parties and radical media and academia. But now Zack Polanski, through his high-profile Green party leadership, regularly questions the reforming credentials of all major Labour figures, including Burnham. Any compromises, retreats or U-turns that he makes as prime minister, in response to rightwing or centrist pressure, will be rapidly highlighted – a big problem for Labour, which needs to win back leftwing voters from the Greens, Plaid Cymru and the SNP.

So how might Labour’s would-be saviour make best use of the hope he has generated? One way would be to demonstrate that he actually enjoys governing, as he did as mayor of Greater Manchester, with his exhaustively publicised enthusiasm for improving its buses. In contrast, Starmer, like the similarly unlucky and besieged Gordon Brown, often made being prime minister appear a burden and a grind. It often is; but many voters already have enough of those things in their lives and may well want a premier who seems less overwhelmed and more excited by the job. Zest in elected office isn’t just presentational – it can provide an administration with a narrative, a sense that diverse policies have a connecting energy, confidence and rationale.

But restoring a sense of hope to British government requires more than setting a tone. Who Burnham appoints, which policies he pursues, and how hard, will show whether his government is a real attempt to redirect a country which, he regularly insists, has been “on the wrong path for 40 years”. As well as Britain’s last half dozen, easy to criticise governments, that period includes the climactic phase of Thatcherism and the Blair and Brown governments, in which Burnham served. Blairite and Thatcherite assumptions, such as prioritising business interests over others, still dominate how much of Westminster, Whitehall and the media – if not so much of the electorate – see Britain’s problems and how they could and should be solved.

In this context, Burnham’s selection of the smooth former Blairite and corporate lobbyist James Purnell as chief of staff is not encouraging for those hoping his premiership will break with the past. For the latter, politically valuable impression to last, this appointment will quickly need to be balanced by less orthodox ones, in keeping with Burnham’s frequently stated belief that Labour should be a broad church.

He will also need policies addressing problems about which many voters have long been feeling depressed and fatalistic. Public control of the terrible privatised utilities; a temporary nationwide rent freeze; the restoration of real power to marginalised towns, regions and cities: such moves would almost certainly be widely dismissed as impractical or dangerous, or both. But if Burnham doesn’t take risks and take on vested interests, his party, the government and the country will sink back into gloom. In disillusioned times, hope can be especially powerful, but the case for it has to be constantly proved.

• Andy Beckett is a Guardian columnist