The benefits budget is now a magic money tree. Whenever Conservatives or Faragists make wild promises – tax cuts, more police, more punishment, more bonuses for marriage – and are asked how they would pay, the answer is always “welfare”. The sums are enormous. “Only the Conservatives will cut welfare spending by £23bn and get Britain working again,” the party insists.

More unexpected was the klaxon from the Labour peer George Robertson this week, demanding a cut in benefits to finance defence. “We cannot defend Britain with an ever-expanding welfare budget,” said the ex-Nato chief, wanting to pluck this juicy plum to fund defence. Good to see him slapped down sharply by the government: there is no “zero-sum game” between these two budgets, said the chancellor’s deputy, James Murray.

“The benefit budget is out of control” risks becoming an accepted factoid, casually thrown by TV interviewers at any passing minister. For actual facts, turn to the Resolution Foundation’s chief executive, Ruth Curtice, whose predecessor, Torsten Bell, is now pensions minister (and who says his views have not changed since moving to government). Bell has always noted that benefits as a proportion of GDP have stayed within 10-11%. “It’s not out of control,” Curtice echoes. “It’s fairly flat when you look at working-age benefits.” It’s pension costs that rise, because of demographics – an increase in the number of retirees – and because the triple lock is designed to escalate pensions regardless of individuals’ wealth or needs. The working-age benefits bill rises with the rising pension age as more older people waiting for their pension, unable to work, draw sickness benefits. Are they scroungers?

For those relishing Kemi Badenoch’s £23bn cuts pledge, Curtice reminds us of the effect of George Osborne’s £15bn cuts in 2015. His two-child limit plunged 450,000 children into poverty and his overall benefit cap added many more. Freezing the allowance for rent propelled record numbers into expensive temporary accommodation. “We feel the consequences still,” Curtice says, “and he didn’t even then manage to cut the full £15bn.” The basic out-of-work rate, at a miserable £98 a week universal credit, is still the lowest among similar countries and 9% lower in real terms than in 2010, despite Labour raising it this month.

Parties championing cuts will keep the expensive and wasteful triple lock, costing an estimated £15.5bn by 2030. They mock Labour for failing to get a £5bn cut through their MPs last year, but that was indeed a bad proposal, a last-minute thoughtless slash in response to the rising number of young people drawing sickness benefits for mental health problems. Other similar countries, says the Institute for Fiscal Study’s Eduin Latimer, spend more on health benefits. So whenever a politician airily promises funds from the magic money tree, demand they say exactly what they would cut and who they would impoverish more.

Stephen Timms, the minister for social security and disability and the most knowledgable social security expert on the Labour benches, has been deputed to review disability benefits. His job, he repeats whenever asked, is emphatically not to make cuts: the Treasury has committed to that. But reform is essential. He celebrates the 7,000 people in his constituency who gained from the abolition of the two-child limit, which the Tories would bring back specifically to fund defence. “The assessment system is dehumanising,” he says. The Resolution Foundation is about to report on the lack of re-assessments for those on sickness benefits because of a shortage of jobcentre capacity. Timms works with all the groups concerned, hoping for broad support for reform, as the education secretary, Bridget Phillipson, succeeded through subtlety and sensitivity in her changes to special educational needs support. The hope, together with the former health secretary Alan Milburn’s review of young people not in education, employment or training, is to get more young people into jobs, despite a bleak employment landscape.

How about that welfare to warfare budget switch? Robertson’s great blast was designed to terrify the government into submission. “We are underprepared. We are underinsured. We are under attack. We are not safe ...  Britain’s national security and safety is in peril.” As the author of the government’s strategic defence review, he protests at the slow response to fund it.

He added the accusation of “vandalism” by “non-military experts in the Treasury”. Now, that misses the mark by a country mile. One thing the Treasury knows very well is the disastrous record of defence spending by those “military experts”. Yet again last year, the National Audit Office (NAO) failed to verify Ministry of Defence accounts.

The list of catastrophic overspending and failures amounts to many billions wasted: their many disaster procurements would fill this page. The £6bn Ajax armoured vehicle, eight years late and likely to be scrapped, is only the latest. The NAO hammered the nuclear programme, with Vanguard-class submarine replacements expected for 2024 now unlikely to arrive before the middle of the next decade.

Robertson himself, as defence minister in 1998, commissioned two white elephants: the aircraft carriers costing nearly twice their starting budget, heavily delayed, too vulnerable to sail, with few aircraft. So when he complains of “corrosive complacency today in Britain’s political leadership” he might question the defence establishment complacency that captured him long ago.

He wrote Labour’s strategic defence review in 1998, and maybe hasn’t adapted to this new world. Delay for cautious defence investment may be wise – along with a pivot towards Europe and away from the US – with new cyber threats and cable-cutting perils, and new protective skills to learn.

One frequently used case for defence spending is that it spurs growth. But that myth was blown out of the water this week by the International Monetary Fund’s latest World Economic Outlook, which warned that it raises inflation and worsens public finances, with only a “modest” rise in output. Spending more on defence is no use unless it is spent well. At least nearly everything dispersed by the Department for Work and Pensions arrives in the pockets of its intended targets.

• Guardian Newsroom: Can Labour come back from the brink?
  On Thursday 30 April, join Gaby Hinsliff, Zoe Williams, Polly Toynbee and Rafael Behr as they discuss how much of a threat Labour faces from the Green party and Reform UK – and whether Keir Starmer can survive as leader. Book tickets here or at guardian.live

• Polly Toynbee is a Guardian columnist