After months of war, the US has struggled mightily to compel Iran to restore stable passage through the strait of Hormuz, let alone accept Washington’s core demands – the abandonment of Iran’s nuclear programme, dismantlement of its missile forces and cancellation of its regional proxy networks. Iran’s military is badly degraded and its regime disrupted, but as of today it continues to prevent most countries from shipping oil, gas, fertiliser and helium through the strait. The global economy is at risk, Donald Trump’s domestic approval is sliding, Russia is profiting, and US military preparedness – particularly in the Indo-Pacific – is suffering.

The US is superior to Iran on every measure of national power that matters. It possesses military forces of overwhelming scale, the world’s largest economy, and the ability to cut nations off from global markets through the power of the dollar. Why has Iran been able to frustrate the US’s designs so thoroughly?

The core problem is that while Trump has claimed to be negotiating, in practice he has relied almost exclusively on military and economic pressure rather than the give and take of real diplomacy. A more workable approach would offer Tehran assurances and incentives substantial enough to make the risks of signing a deal with Washington worth taking. And it would respect the red lines that the regime has showed it will not budge on.

Trump’s approach is a form of coercive diplomacy. Coercive diplomacy can and has worked in the past. But it requires making demands the adversary can meet without jeopardising its own survival. That, for example, was the logic behind the coercive diplomacy that brought the Serbian leader Slobodan Milošević to the table over Bosnia in 1995 and to terms over Kosovo in 1999.

With Iran today, however, Washington’s demands have verged on a call for unilateral disarmament. For Tehran, accepting these would mean surrendering the very defences the regime believes protect it from being toppled. Paradoxically, the more Washington escalates military pressure, the more Tehran is likely to conclude that stronger deterrent capabilities – including maintaining some control over the strait – are essential to regime survival. Trump’s withdrawal from the Iran nuclear deal in 2018 meanwhile underscores for Tehran that Washington might just pocket any concessions and return to hostilities.

Iran today also has more capacity to absorb the pressure than most past targets of coercive diplomacy. Drones, missiles, cyber tools and information operations give it instruments to harass and threaten US regional assets, US allies and global commercial shipping. More important, Iran has powerful external supporters. China is providing critical economic and diplomatic support, while Russia continues to offer military and political backing.

Breaking the stalemate therefore means a more realistic negotiating position that accepts that the US bottom line cannot be effective Iranian disarmament. No Iranian government can agree to that and expect to survive. As former European negotiators have pointed out, any substantial deal on the core issues will probably also require near-term sanctions relief for Iran substantial enough to make the political risks of concessions worthwhile for Tehran. Finally, Iran will need some hope that Washington will honour the deal it signs rather than pivot back toward regime change. The participation of third parties – China, Europe, possibly the Gulf states – would help achieve this.

Getting more flexibility in the US position will be tough, however, especially because US allies in the region will resist sanctions relief unless there are major Iranian concessions on the nuclear and missile fronts. But the alternative is a continued stalemate in which Russia profits, China’s leverage grows, allies in the Indo-Pacific watch American resources drain into another Middle Eastern war, and the global economy risks recession.

If a negotiation on the major issues proves impossible, the realistic floor is to negotiate a return to prewar freedom of navigation through the strait, and a freeze on further military escalation. This may be the direction the administration is headed, based on recent press reports. Trump can try to sell the damage done to Iran’s military and nuclear infrastructure as a victory for US interests. In reality, of course, this would not be a success. But it would stop the erosion of American power that this war has caused.

Trump’s bind in Iran is the predictable outcome of the conceit that overwhelming military and economic power can substitute for the willingness to compromise. It is a conceit that has repeatedly produced strategic disappointment for major powers throughout the post-cold war era – from Iraq to Ukraine – proving yet again that military power is no substitute for real diplomacy.

• Christopher S Chivvis is a senior fellow and director of the American Statecraft Program at the Carnegie Endowment for International Peace