Three years ago, in a greasy spoon on the fringes of the City of London, M John Harrison – Mike to his friends – told me about the novel he was working on. Rather than describing its plot or characters, he spoke purely about the challenge the book presented to him as a writer. With this one, he said, he wanted to push things as far as they could go.

Now that book, The End of Everything – his 13th novel – is about to be published. It describes a disintegrating Britain in which the iGhetti – monstrously sized, extremely powerful and strange lifeforms that look like powdery, slow-motion explosions – rule the country and possibly the world. Or do they? In its unwillingness to divulge any more than its characters know, which isn’t much, the novel is more alien evasion than invasion.

No one knows where the iGhetti came from. Maybe the astral plane, or “out of the internet”. Their purpose is similarly obscure. What remains of the authorities treat them as hostile, sending ineffectual waves of bombers and attack helicopters, but the incomers might equally well be engaging in “spiritual tourism and gentrification” as in colonisation. “If we were to meet a real alien,” Harrison says, sitting on the sunny terrace of a riverside pub in Barnes, south-west London, where he used to live, “we would have no clue whatsoever what they quote ‘thought’, or why they did anything, or if they thought they were doing something.” Science fiction often pays lip service to that idea, he says, but “never passes that feeling on to the reader”.

Harrison is a slim, nimble 80-year-old, his full beard and long hair glowingly white. His skin has the nutty tone – unusual in writers – of someone who has spent plenty of their life outdoors. The planes of his face look austere in photographs but in person he is often laughing, and the eagerness with which he talks about meeting the demands of the new book underlines how much he’s enjoying himself.

This wasn’t always the case. In 1998, a year after Harrison published the bleak toxic waste-themed dystopia Signs of Life, Iain Banks took him out for a night of drinking in Soho with the aim of persuading him to return to the purer sci-fi realms where his career began. “I always keep in mind what Iain said to me,” Harrison admits, “which is that I don’t have enough fun on the page. That was hurtful.” The next day, he started writing the notes for Light, the first volume in his Kefahuchi Tract trilogy. Not the space opera Banks suggested but a parody of one, because nothing with Harrison is straightforward. “Nothing at all,” he happily agrees.

Harrison was born in Rugby, Warwickshire, in 1945. He had a difficult relationship with his father, an engineer, who died when Harrison was 13. He truanted a lot from school, spending part of each day in the local library. “The great thing about libraries then was there weren’t so many dust jackets about,” he tells me. “I would pick a book up, read the first two pages, think, ‘Oh wow, that’s weird’, and it would turn out to be a Robbe-Grillet, and it would open a door to the anti-novel. Or it would be Ballard, or another sci-fi book. You never knew what you were going to get.”

When Harrison began writing as a teenager in the 1950s, the genres of fantasy and sci-fi were supported by numerous monthly magazines. In 1966, one of them accepted a story of his. He moved to London and began writing obsessively through the night. He met Michael Moorcock, then the editor of New Worlds magazine, and became a regular contributor. “I had to be in New Worlds,” he says, “because it was Ballard’s main medium for short stories at that time. It was at the height of my interest in him as a kind of combination of a surrealist and an imagist. Especially in the short story form. And I wanted to be that. I really wanted to be that.”

On his blog, Harrison has described The End of Everything as the kind of book that might have been serialised in New Worlds circa 1967. I’m not so sure they would have accepted it. “I think it might have been too much even for them,” he agrees. “I wanted it to have the flavour of the novel that I would have submitted then if I’d had any technique, skill or talent, a book that on the surface looks like sci-fi but then, as you read it, gets depthier and depthier. That was what I wanted. My heroes could do that. And now, 60 years later, so can I.”

He laughs as he says this, but it’s taken Harrison a long time to arrive at a place where he’s happy both with his work and its reception. The 1970s saw him strain against the genre conventions of sci-fi and fantasy, which he tried to undermine in The Centauri Device (a book he now disparages) and his Viriconium sequence. A breakthrough occurred when he resolved to write a short story without allowing himself to plan it out beforehand or keep notes. The New Rays is “about Katherine Mansfield. And it’s for Katherine Mansfield.” He admired what she and Virginia Woolf had done with fragmented narrative back in the 1910s and 20s (Eliot’s The Waste Land was also formational) but didn’t know how to approach that way of working himself. “The only techniques I had were almost exactly opposite to what I needed. They were the techniques of genre fiction: making a narrative, making a synopsis, following the synopsis, making the causalities plain, following the causalities. And none of that would do.”

By the time The New Rays was published in 1982, Harrison had left London for “the boondocks outside Huddersfield” to pursue an obsessive interest in rock climbing. The next two decades, which saw the publication of the novels Climbers (1989), The Course of the Heart (1992) and Signs of Life (1997), were the most intensely creative of his life. “I let it take over,” he says now of writing. “And I produced, as a result, several short stories and three novels that had real depth and density of observation, and a deep, dense sense of place.”

This is an understatement when it comes to Climbers, which isn’t just one of Harrison’s masterpieces but one of the best English novels of the last 50 years. The book follows a group of climbing junkies around the Peak District, men and women who, like many of Harrison’s protagonists, are out of joint with the wider world. It is still criminally obscure despite the loud enthusiasm of, among others, Robert Macfarlane and Olivia Laing.

As we quit one pub for another, walking through the quiet Barnes streets, Harrison recalls the moment the book became possible. Leaving a quarry outside Sheffield at sundown one day, “I noticed that the way the sun related to the jagged top of the quarry, from my viewpoint, meant the shadows looked like the turned-down pages of a book. I stopped and scribbled that in one of my notebooks. I suddenly thought, I can do this. I’m the person to do this. It was really weird. What had stopped me from writing fiction about my own experience, or even nonfiction, was that I didn’t really feel I was the person to do it. I didn’t feel I had the authority. And then I wrote that sentence down, looked at it and thought, ‘Yeah, I can do this.’ It was amazing,” he says, still sounding as surprised as he was in that shadowed Yorkshire quarry decades ago. “You hunt for that your whole life.”

Harrison, “utterly determined to stop apologising for not being an SF writer”, now began producing work he fully believed in. But he had – and still has – an uneasy relationship with his creativity. “It was like discovering a different voice inside you,” he explains. “And it was better than me. I’m going to tell you this,” he says, lowering his voice as if this other presence might hear us. “He knows more than I do, he’s more mature than I am, he’s a better writer than I am, and he has very considerable contempt for me. But every so often he’ll look at something and think, yeah, that’s OK, and he’ll step in and take over and produce something like Climbers.”

At times, Harrison says, he feels he is the impostor. “There are two of us and one of us knows he’s the real me, and it isn’t me.” Then, thankfully, he laughs, dispelling the eerie sense of having slipped into one of his own fictions, where terrible things are revealed in the most pedestrian surroundings: a Pizza Express, a drab provincial courtroom, or a pub in Barnes after the lunchtime rush.

Having moved back to London after realising he was too old to continue climbing (and perhaps because some in that community “were offended by the clarity of the portrait”), in 2012 Harrison found himself suddenly overcome with anger at a publishing party in Covent Garden. “I got outside,” he says, “and the rain was pissing down and I flashed back to 1968: same street, same rain, same sense of failure, same sense of not getting on with the industry.” He remembers thinking: “I’ve wasted 30 years of my life in London and I’m no further forward. I’ve learned all this stuff and I can do all these things and it’s still not been recognised.” The solution, he thought, was to “be even more uncompromising in the provinces”.

He moved to Shropshire with his partner, the editor and writer Cath Phillips, and started to write The Sunken Land Begins to Rise Again. That book won the Goldsmiths prize in 2020. Frances Wilson, the chair of judges (of which I was one), called it “a literary masterpiece”. Harrison remembers the ceremony, an online affair due to Covid restrictions. “I felt so relieved. I had a drink or two and fell asleep. I relaxed for the first time in about 40 years. I thought: ‘I won a proper prize. I can go to sleep now.’”

The work of most writers who publish into their 70s and 80s tends to decline in quality. With Sunken Land and The End of Everything, and his “anti-memoir” Wish I Was Here, Harrison has produced some of his best work. One reason climbing was such a perfect subject is that he is motivated by problems, and climbers view a rock face as a sequence of problems.

The problem presented by The End of Everything, the one he talked about in that greasy spoon in 2023, was how to leave so much out while still exploring how “human beings are working with broken epistemologies to try and understand the world that we’ve made. The enigmas of reality,” he explains, “as in, say, quantum mechanics, aren’t the real mysteries any more. The real mysteries are what the fuck we’ve done to the world, why we did it, and what epistemology we used to perform this act of vandalism.”

Conveying bafflement without sacrificing readability is Harrison’s recurring problem, one he’s faced “for 30 or 40 years. You’ve got to be so careful with explanation,” he says, sounding almost pained. “If you help the reader too much, you lose that inexplicability. You’ve got to commit.” The End of Everything is the result of that commitment, thrilling to experience because, not in spite of, its resistance to disclosure.

The book is dizzying in its invention – not only in Harrison’s creation of a post-invasion world of semi-abandoned seaside towns, crashed airliners and repurposed polytunnels, but also at the granular level of moments you want to return to, sometimes for the sake of comprehension, sometimes just to re-experience their strange power: the “clean arch of brand new stars” revealed after the iGhetti’s arrival; the “rich surf of objects” – alien detritus – his characters scavenge from the sea. It is also a continuation of that late-night Soho conversation from nearly 30 years ago. “I thought: OK, here you go, Iain,” says Harrison. “I’m having fun but I’m also gonna commit. This is gonna be the one that is written without any compromise.”

And if the title sounds ominously final, we shouldn’t read into it. “I’ve got two or three short stories which,” he says with relish, “are being very intractable.” On to the next problem then? He laughs. “Yeah, what’s the next problem? What impossible thing can I try and do now?”