Simone Stolzoff describes himself as “naturally an uncertain person” inclined to rumination and self-doubt. This tendency benefits him in his work as a journalist, but can otherwise be a double-edged sword.

While working for a magazine in New York, Stolzoff was approached about a job at a design firm in San Francisco. Now, he laughs at how tortured he felt “having to decide between two attractive career paths”.

But, at the time, “it really sent me for an existential loop,” he says. “I could see these two diverging paths – Simone the journalist, Simone the designer – and, for the life of me, I could not make up my mind.”

Stolzoff talked his options through – with everybody. “My yoga teacher, my Uber driver, all my friends and family …” He grimaces. “I was insufferable.”

He wound up choosing the San Francisco job: a new home, an unfamiliar industry. The experience informed his new book, How to Not Know: The Value of Uncertainty in a World That Demands Answers.

Looking back, Stolzoff says over video-call from his home in the Bay Area, his mistake was aspiring to feel certain: “It was my intolerance of uncertainty that was causing so much of the angst.”

Trying to anticipate the future and make plans with confidence is unique to humans, and evolved to keep us safe. But, especially in these uncertain times, it can easily lead us astray, says Stolzoff: “We have these brains that are wired to get out of uncertainty as quickly as possible, in a world where there are triggers all around us.”

Because the future is unknowable, the real question is how we can learn to cope better with not knowing, Stolzoff says. Our conversation has been condensed and edited for clarity.

Why did you write this book?

My first book, The Good Enough Job, is about how work came to be so central to our identities. The most common question asked by readers was “How should I think about the future of my career, given AI and all these other changing forces?”

The honest answer was that I wasn’t sure. I wanted to throw my hands up and say “I was a poetry student, for God’s sake”. That is not a very gratifying response to receive, so I wanted to explore this question of uncertainty, and not just insofar as it relates to career.

The cliché is that you write the book you need to read – I went on this multi-year journey of exploring the science and psychology of uncertainty partly so that I could hopefully get better at dealing with it myself.

The modern world is often said to be acutely, even uniquely turbulent, but we also know more about it than ever. Why might we be struggling in this moment?

It is a precarious time. Stanford economist Nicholas Bloom has been studying global uncertainty since the early 1980s; the five highest measurements all occurred in the past five years.

What I think is novel is the fact that our tolerance for uncertainty is in decline. Research has found a correlation between this and the rise of the internet, and more particularly smartphones. These pocket computers create the expectation that answers should be readily available. Ten years ago, I might have been okay with not knowing the name of some actor; now I feel an almost involuntary need to find out right away. They also bring all of the world’s uncertainties to the fore.

But often these real-time updates and access to information just fuel our anxiety, and rob us of practice in sitting with what we don’t know. The best way to increase your tolerance for uncertainty is through exposure, and resisting easy answers, but the internet makes it much easier to avoid that discomfort.

Are there benefits to becoming more comfortable with uncertainty?

Many people are unwilling to face uncertainty head on. They might know their job or relationship isn’t working for them, but would rather stick with the devil they know. But by being willing to embrace the unknown, you may discover something that was greater than what you could have anticipated.

I hope people come away from the book understanding that, yes, uncertainty can be threatening, but it is also the birthplace of possibility. If you’re willing to tolerate it for a bit longer, you can discover new things about yourself, others and the world.

There’s also a clear mental-health case: our brains tend to think about the worst-case scenario, and catastrophise. A lot of our anxiety is driven by worrying about things that haven’t happened yet. I think there would be a lot of collective health benefits if we could be more okay with what we don’t know.

Uncertainty can register as a threat, triggering the fight-or-flight response. Should we think of uncertainty intolerance as a physiological issue, as much as a psychological one?

We spend so much time up in our heads – particularly knowledge workers, like you and me – but a lot of wisdom is embodied. I think often our brains are catching up. Research shows when you’re able to regulate your nervous system, you’re literally able to see more possibilities, but when you are in that fight, flight or freeze mentality, you’re not necessarily in that grounded place.

Many psychologists told me of the importance of slowing down when you’re facing uncertainty, in order to go from your reaction brain into your more analytical brain. Then you’re able to assess the different possibilities, as opposed to just acting to avoid the threat as quickly as possible.

Part of the issue seems to be that reality itself feels uncertain right now; we don’t necessarily feel able to trust in what we felt certain of 10 years ago.

Totally – there’s no toehold, we don’t have a common basis of shared truth. On the one hand, we should be more sceptical in this age of chatbots, deep fakes, et cetera. On the other hand, how can we find common truth, and not just exist in our bubbles?

I do think that intolerance for uncertainty is at the root of so much of our political polarisation. I think our world would be much more connected if we didn’t jump to conclusions – about exactly who someone is, based on who they voted for.

Even the loneliness epidemic: you have to be willing to enter into an interaction with a stranger, not knowing how it will go. We have to put ourselves out there, because that’s how we develop the [internal] data and evidence that uncertainty is not necessarily something to fear.

I was struck by the quote from psychology professor Philip Tetlock, who analysed 20 years of public predictions and found the average expert was “roughly as accurate as a dart-throwing chimpanzee”. What do we typically get wrong about the future?

The psychologist Daniel Gilbert has this concept called the “end-of-history illusion”. We often assume that who we are today will be fixed. It’s hard to recognise that what we might be doing for work in ten years – or who we’ll want to date, or what we’ll like to eat – might be very different from what we’re doing now.

That gives me some solace: recognising that I’ve faced lots of uncertainty in the past, and come out the other side. Often we discount our ability to course-correct or adapt. Some decisions, such as “should I buy this house?” or “should I marry this person?”, deserve a more deliberative thought process, because it’s harder to go back on them, but most are somewhat reversible.

There’s a huge cost if we take that highly analytical framework and apply it to decisions like what to watch on Netflix. Part of my goal is to help people not see uncertainty or doubt as something that should prevent them from making choices or acting.

My takeaway from your book was that action is the best response to uncertainty – rather than waiting for an unequivocally correct course, or giving into paralysis.

The metaphor that I come back to is rowing through the fog: you might not be able to see very far ahead, or know exactly where you’ll end up, but you have to keep rowing.

None of us have perfect information – we’re just doing the best we can for this version of us, at this moment in time. I think of it as: how can you make a decision that reinforces the type of person that you want to be? If you act in alignment with your values, you can still stand by the choice, even if you don’t get the outcome that you desire.

At the same time, I’m not trying to tell people to seek out uncertainty. Some certainty makes it easier to hold uncertainty in our lives, even if you are feeling very anxious about the future. One of the most practical pieces of advice in the book is to identify your anchors, the things that will remain constant through the changing winds. I think about my family, my values and my commitment to my home.

What did you take away from your time in the tiny Pacific nation of Tuvalu, at imminent threat from rising sea levels?

That was the most fulfilling reporting for me. The climate crisis is literally lapping against the shore. I felt so inspired by how Tuvaluans are approaching it. One of my interviewees was a homesteader, focused on how to become more self-sufficient. Another was an international diplomat, wanting to instill more collective reliance. They’re two approaches to uncertainty: it’s not either-or; it’s both-and.

It’s a nice analogue to so many other crises that we’re facing – AI and jobs, for example. You might want to insulate yourself from the coming technological disruption, get really good at using the tools and think about how AI might improve your workflow. Or you might build collective solidarity and resilience through your network and relationships. Both approaches are good. Often they’re set up in the media as opposites: you’re either Team AI, who thinks technology is going to allow you to focus on higher-level tasks, or Team Anti-AI, who thinks the robots are coming with pink slips. I think the truth is probably somewhere in between.

You suggest that intolerance for uncertainty is related to fear of death – but that both are essential to the human condition, and part of what makes life special.

Reckoning with death helps us get clear on how we want to live. The cost of our tendency to shield our eyes from our mortality is that we aren’t able to see the preciousness of life in its finitude. As much as people want to follow longevity protocols and live forever, I don’t think that’s actually the best way to live a meaningful life. Part of what makes life meaningful is the fact that it’s not going to be forever. If we had certainty about exactly when, or how, we were going to die, I think that would be a bad thing.

In general if you want change, or to make a difference, you shouldn’t have a fixed idea of what the world will look like in 2050, because that’s disempowering – it removes your agency to be that change. In the uncertainty, that’s where magic, surprise and delight lives.

• How to Not Know: The Value of Uncertainty in a World that Demands Answers by Simone Stolzoff is out now