In the churchyard next to Wilmington Priory in East Sussex, I found a yew so ancient and stooped that its trunk had eaten half a gravestone. Its boughs were supported by long poles, a creepy sight that made me shudder. I had come here to see something just as strange, but more benign than this folk-horror vision – the figure of the Long Man of Wilmington on the hillside opposite, on the steep scarp of the South Downs. He treks over the hill, a stave clasped in each hand. Climbing Windover Hill, just beneath the South Downs Way, I saw that while he was once a chalk giant, his lines are now marked with concrete blocks.

The Long Man may be Anglo-Saxon in origin – the shape is similar to the design on a buckle discovered in Kent in 1964 by the archaeologist Sonia Chadwick Hawkes, which probably represents the god Odin (or Woden); but he may be a much later adornment for the hillside, made to be viewed from the priory. His form entranced the photographer Lee Miller and her husband, the artist Roland Penrose, who lived close to the Long Man. Penrose painted a surrealist representation of the Long Man on the inglenook fireplace at Farleys, their home – for them the figure was a protective spirit. It also inspired the composer Avril Coleridge-Taylor, the folk collective the Memory Band, and Benjamin Britten picnicked at its feet.

The Long Man of Wilmington is one of the more famous chalk figures, the mysterious carvings that decorate the hills in England (and almost uniquely in England), numbering 40 or so. They have always been a part of my life. Familiar and simultaneously fantastic, they have fascinated me and many others: film-makers, writers, musicians and artists. They have drawn the attention of historians, archaeologists, antiquarians, all sorts of fellow travellers. Their appearance enlivens walks and invites conjecture.

Many hill figures – most famously horses, but also crosses, crowns, regimental symbols, giants and buried gods – are located close to ancient trackways that have taken pilgrims, traders, warriors and now Gore-Tex missionaries over the rounded chalky hills. By tracing these routes for my book The Tattooed Hills, I was able to get under the skin of these mysterious shapes.

My planned walk of a few miles from the Long Man to the figure of the Litlington White Horse was cut short by a thunderstorm – you don’t want to be in the hills under lightning – so I visited the next day. This small, lonely animal peeks over the hill towards the English Channel near Cuckmere Haven, and was cut secretly, by local people, in a single night in 1924, the successor to an earlier lost figure.

Also close to the coast, in Dorset, I climbed the hill to the huge figure of George III on his horse, Adonis, overlooking his favourite seaside resort of Weymouth. It was carved in 1808 as a tribute to the king and a huge advertisement for the town, although he’s rather faded now. Walking farther north, along the Wessex Ridgeway, I was menaced and fascinated by the huge, priapic figure of the Cerne Giant, on the opposite hillside, dominating the secluded valley with his club held aloft. His date has been hotly contested – he has been believed to be a Romano-British figure, or a 17th-century marauding Oliver Cromwell. Some have thought he was Helith, a pagan god. He is more likely to be a Saxon image of Hercules, or a local saint, Eadwold. This giant is an unreliable shapeshifter, a joker.

Taking the Ridgeway across Wiltshire’s chalk hills will plunge you into the county of white horses, of which eight remain. A midsummer walk from the slim form of the Alton Barnes White Horse took me along the Wansdyke, a great defensive ditch and bank stretching for miles through empty countryside with views of the ancient Silbury Hill, to a similar horse at Cherhill, scampering over the deep-sided coomb, carved in the shadow of a hillfort and signposted by the Lansdowne Monument, which dominates the hillscape and makes it unsettling.

Both horses come with music: the Alton Barnes horse appeared briefly in a video for the Britpop band Dodgy, for their single Staying Out for the Summer. The song mashes together many nostalgic cliches of the time (VW camper vans, space hoppers, crop circles, football tops), summoning up a specific idea of the countryside as a place for raves, and selling it back to us, without its rebellion. In their earlier incarnation as the Timelords, the KLF came to Cherhill to film the video for Doctorin’ the Tardis – not a good song, but one which introduced us to their art-terrorist antics. The KLF make the horse part of a dustier landscape in keeping with this corner of Wiltshire’s weirdness.

Further along the Ridgeway, I came to the greatest and most mysterious figure – the commanding presence of the elongated Uffington White Horse, also sited beneath a hillfort. Archaeology has dated this figure to the late bronze age – 3,000 years ago, give or take – and it’s an extraordinary survival. Generation after generation have cared for this racing animal, somehow keeping it bound to its wind-blown hill. It too has been an inspiration for musicians: Kate Bush’s Cloudbusting video was filmed here and XTC’s album English Settlement has the horse on its cover. Andy Partridge from XTC told me that the landscape surrounding his native Swindon was an important inspiration: “It marked me like an Avebury stone or the ripples across a hillfort. It made me.”

The Ridgeway gets tangled up with the Icknield Way, which runs over the country’s chalk spine to East Anglia and through the leafy Chiltern Hills, home to a cluster of some of the stranger chalk figures, which include two crosses, one on top of a massive chalk pyramid – the Whiteleaf Cross. Another figure, the Watlington White Mark, has been interpreted as an ancient fertility symbol, but is actually an 18th-century trompe l’oeil of the landscape. Stand in a particular place, it is said, and the chalk obelisk gives the church a spire. Walking east along the Icknield Way took me to Ivinghoe Beacon; what drew my eye was the magnificent chalk figure of the Whipsnade White Lion, cut in 1931-33 to celebrate the opening of the zoo and now occasionally nibbled by wallabies, who help keep it in good shape.

At the far end of the Icknield Way I visited Wandlebury, in the Gog Magog Hills, named for the giants who, in folklore, once ruled Britain. One archaeologist, TC Lethbridge, thought he had found hill figures beneath the turf here; his findings were speculative and nothing remains of them, but the iron age hillfort above, the beechwoods and the nearby ancient track are the chalklands condensed into one country park. What struck me most about the Gog Magog story was something Lethbridge wrote in his book describing the dig, something that I found too as I travelled in search of the stories of the chalk. “There is no need,” he wrote, “to go to the ends of the earth for interesting quests and excitement. It is here, in prosaic old England, at one’s back door.”

The Tattooed Hills: Journeys to Chalk Figures by Jon Woolcott is published by Aurum (£17.99). To support the Guardian, order your copy at guardianbookshop.com. Follow Jon on Instagram at dorsetjonw