‘Finn! Finn! FINN!” Johnnie Shand Kydd is having trouble keeping his inquisitive lurcher in sight. Finn may be an incredibly sweet-natured dog but he’s hard of hearing – and has previous for disappearing on this particular walk.

At least the photographer has experience in dealing with unruly characters. In the 1990s, he found himself embedded with the Young British Artists, granted free rein to shoot the hedonistic, chaotic and wildly creative art scene that birthed Tracey Emin, Damien Hirst, Sarah Lucas and more. Shot in black and white, these images upended the convention for artists posing in their studios, easels in hand. “I just wasn’t interested in that at all,” says Shand Kydd. Instead, his photographs capture Hirst balancing a tower of hats on his head, Emin in a rubber dinghy with Georgina Starr, a newly pregnant Sam Taylor-Johnson (then Taylor-Wood) and a whole load of partying, boozing and canoodling.

Shand Kydd wasn’t a photographer when he began documenting the scene – he claims he’d barely taken a picture before in his life – but he was a former art dealer who understood artists and their delicate balance of ego and insecurity. The YBAs quickly relaxed in his presence. “It really wasn’t very hard,” he says with a laugh, happy to play his talents down. “Taking a photograph is the easiest thing in the world. You just point and click. It’s finding something to say that is the hard bit.”

Well, that and gaining your subjects’ trust. Many of his YBA images were taken during gargantuan benders, which led to Shand Kydd amassing a fair few compromising images, not that he ever considered publishing them. “Why ruin a friendship with these incredible artists for the sake of a yet another photo?” he says. A more pressing issue for the photographer was keeping up. “I was a wee bit older so I didn’t quite have the same stamina. If they were off on a 48-hour bender, I would probably duck out after 24 hours. Although often you’d just be about to leave and then the double doors of the Groucho would open and the next 20 people would come in.”

His images, published in the 1997 book Spit Fire, no doubt fuelled some of the stuffier criticism that these brash, gobby upstarts were unserious people who favoured hedonism over hard work. Shand Kydd, though, was perceptive enough to realise that nights out like this were actually being used by modern artists as an extension of the studio. “You would always see people in a huddle, having these really intense conversations in the early hours of the morning. This was about ideas being exchanged. Besides, I’m sure David Bowie can barely remember making some of his best albums. There’s a certain moment where youth, talent and inebriation all come together.”

And there was a lot of hard work involved. “They didn’t need expensive materials, expensive galleries or anything like that,” he says of the fertile creative environment. Whenever he visited Hirst’s studio in Stroud, Gloucestershire, he would return feeling particularly inspired. “The energy there was exhilarating. Damien was a firm believer that the word ‘No’ didn’t exist. Everything and anything was possible. It was the optimism of the period that I remember most.”

Now 66, Shand Kydd is still in touch with his former subjects. He was recently out for lunch with Lucas – “you can bring up any subject and she will approach it from the most unusual angle” – and Taylor-Johnson is apparently making a film based around one of his images. Emin, meanwhile, curated his new exhibition Ramsholt, currently showing at her gallery in Margate and soon to be published as a photobook. Named after the tiny Suffolk village his mother moved to in the 1960s (she still lives there – on the day we meet, she is celebrating her 93rd birthday), Ramsholt features photographs taken on Shand Kydd’s dog-walking route with Finn and Finn’s daughter Zelda, who is sadly not well enough to join us today.

Ramsholt looks beautiful in the April sunshine: fluffy clouds bounce off the surface of the River Deben while the trails are abuzz with St Mark’s flies. Beaming sunlight is not, however, Shand Kydd’s preferred condition for capturing the unvarnished and often bleak scenes you’ll find in his book: barren fields, mist-covered trees, uprooted trunks. The director John Maybury told him the images looked like “crime scenes”. When Emin first saw them she said: “Your photographs are all about death.”

Did he agree? “I did. Because with nature you can’t have spring without winter. And fungi growing on rotten wood – it’s all part of the reincarnation element. One of the pictures she liked the most is of daisies. She just said, ‘That’s what happens after we’ve left.’ I thought that was very optimistic.”

Having become accustomed to taking photographs of other people, Shand Kydd believes these images are by far the most personal things he’s ever put out there. In an essay to go with the book, he touches on past family tragedies: the uncle who died in his teens performing aeroplane stunts, and the damage this may have done to his father’s relationship with Shand Kydd’s elder brother, a novelist who died in Cambodia in 2004.

He also provides a fascinating history of Ramsholt itself, flitting across eras and stories in a way that, he hopes, mirrors daydreams you might have on a walk like this. We hear about the American B-17 Flying Fortress that crashed in 1945, killing eight passengers (two survived), as well as the location’s more recent attraction to Ketamine smugglers.

The point he’s making isn’t that Ramsholt is an especially interesting place. “It’s really rather ordinary,” he says. Rather, it’s a reminder that beauty and adventure exist everywhere, if we make the effort to look. He points at a delicate leaf hanging from a tree to illustrate his point. No camera with him to record it today though.

In fact, on most of the walks that inspired his new book he neglected to bring a camera – the unwieldy large format ones he likes were “too heavy” when he had two dogs in tow. If he saw something special, he would go back and photograph it later. “Basic spontaneity was thrown out of the window,” he says, laughing. “The total opposite of how most photographers work.”

But Shand Kydd has always gone against the grain. Realising photography could be his metier after the success of Spit Fire, he decamped to Naples to properly teach himself, the resulting images appearing in his 2009 book Siren City. But his self-deprecating manner means he’s more likely to dwell on his gaffes than his successes – like the time he was flown out to shoot Madonna at a UN gathering in New York and missed his moment because his camera was still in his back pocket when she whizzed past.

It was at this event, surrounded by celebrities and feeling somewhat awkward, that Shand Kydd found himself chatting to “the only other person there who was as equally out of their depth”. After chewing the fat for a while, he thought it would be rude if he didn’t offer to take a picture of this guy and his wife. “I thought nothing of it, until years later when he becomes president.” He laughs. “I went back to the contact sheets and sure enough it’s Donald and Melania.” What was he like? “A few pence short of the pound, but fairly affable. But then I thought P Diddy was incredibly charming so I’m no judge of character.”

We’ve been walking for well over an hour and have more than earned our scampi and cider lunch at the Ramsholt Arms. Outside on the wall is a plaque commemorating the victims and survivors of the B-17 crash. Life and death. Things do seem to collide in Shand Kydd’s work. When his father remarried, he became stepbrother to Charles and Diana Spencer – they would holiday and play together (he is still in touch with Charles).

He remembers how he was working on the installation of Sensation – the iconic exhibition at the Saatchi that helped propel the YBAs into the stratosphere – when he heard Diana had died. “I could walk over from the gallery to where people were mourning,” he says. “So the two things are very intertwined for me.” On reflection, he sees the parallels between Diana and his former subjects. “They both liked lobbing a grenade into proceedings,” he says with a smile, “to see what might happen.”

A decade after Diana’s death, Angus Fairhurst, a YBA who appears in Spit Fire, killed himself while out walking in Scotland. “I didn’t realise he was in such a dark place,” says Shand Kydd. “One thing I’ve learned is that you can never really know what’s going on in other people’s minds.”

He believes Fairhurst’s death caused a lot of his peers to slow down – to grow up, take stock and focus on what really mattered: the work. On a more personal level, though, Shand Kydd credits Finn and Zelda with keeping him on the straight and narrow. “If you’re wasted in front of an animal,” he says, “it can really freak them out.” Such is his devotion to his dogs that he won’t even leave the house for the evening unless he can find a sitter.

“He really is a lovely boy,” he says, looking over at Finn, who is close to falling asleep on top of his coat. Yet another subject who feels totally trusting of their master.

Ramsholt is published by Cheerio on 7 May