Disabled people who use blue badges to go about their daily lives have said they are being harassed, questioned and even assaulted, as anti-benefits rhetoric becomes more mainstream in the UK.

About 3 million people in the UK now have a blue badge, including 1 in 15 adults in England. The number of people who qualify for the scheme – which allows drivers to park in more accessible spaces – has caused some to warn of misuse and fraud.

The AA has called for a crackdown on people using fake or stolen blue badges as the number on the scheme grows, while the Daily Mail “names and shames” drivers taken to court for fraudulently using a badges.

But the culture of suspicion has, according to the dozens of users who contacted the Guardian, given rise to a tide of abuse from members of the public towards badge holders, including accusations that they are faking their disability.

One badge holder, Charlotte* from Aberdeen, has not been back to her local Co-op since her carer was assaulted after parking in a disabled bay last year.

“I was helping my carer put my shopping away in my car and a lady came up to my carer and said: ‘You don’t look very disabled to me, you shouldn’t be parked there.’ My carer told her the badge is for me. The lady got so angry that she pushed her trolley into my carer.”

When Charlotte, who has muscular dystrophy and autism, started crying and got back in the car to leave, the woman then used her vehicle to block them in, she said, before using her vehicle to block them in. “She was waving her hands around, swearing, just so angry,” the 27-year-old recalls. “We’ve never been back since. I’m worried my carers will get assaulted again just because they’re helping me park somewhere I need.”

Disabled people using accessible services have long faced harassment from other members of the public but there are fears the rise in rhetoric about “free BMWs” and “golden tickets” in recent months has legitimised the questioning of people’s disabilities.

The Guardian has heard from almost 100 disabled people and their families who have been challenged by strangers for using a blue badge. Many said they have been verbally abused, with some being filmed on phones, physically intimidated, or having their mobility aid pulled away.

Recent research by MS Society showed more than one in five people with multiple sclerosis in the UK had been questioned for using a blue badge, with some reporting being shouted at, harassed or even spat at.

“On a weekly basis, I get challenged by the elderly and experience hostility for using my blue badge, because I look young,” said Dave, 35, from Bournemouth. “Some make sarcastic remarks such as: ‘Using your nan’s badge?’”

Dave, who needs a walking stick due to fibromyalgia, has been repeatedly recorded by members of the public when he parks in a disabled bay. “In February, an elderly man started filming me on his phone and said: ‘You’re clearly not disabled, I’m putting you on social media.’”

On another occasion, an older woman filmed him through his car window while physically stopping him from getting out. “She had such a menacing face, so I opened the car window and she said: ‘Move your car, I need that spot. I’m disabled, I’m elderly, you’re not. You’re clearly using a fraudulent badge or someone else’s.’”

Emma*, 45, from London, said public hostility towards her disabled child meant she now found it hard to leave the house. Seven-year-old Amelia* has a severe learning disability and needs a specialist buggy due to mobility problems. When the family used a blue badge space in a multi-storey car park last year, an older woman questioned whether Amelia was really disabled. “This particularly upset her [11-year-old] older sister, who is very protective,” Emma said. “I just wanted to cry.”

Emma has anxiety and depression and now finds herself “tensing up” when the family goes anywhere. “Facing harassment and scepticism about our daughter’s blue badge entitlement makes it harder for us to go out and means we’re more isolated,” she said. “It’s a case of working out if I’m strong enough on the day to deal with [it].”

“I feel like all the latest stories about [disability benefits and entitlements] have encouraged others to confront people with blue badges who don’t fit what’s typically seen as disabled,” said Sarah, 41, from North Yorkshire, who uses crutches due to osteoarthritis and is waiting on a double knee replacement. “People do get in my face and they are aggressive. I’ve had people try to take my crutches off me, saying I don’t need them.”

Sarah said she had been accused again last week of faking her disability, when she used a blue badge at the supermarket. “I showed the security guard the back with my picture on but the person making the claim followed me around [as I tried to shop] saying I’m too young to be disabled or that I have both legs and I’m not in a wheelchair,” she said.

The increasing harassment means Sarah rarely leaves the house now. “I’m scared someone may actually attack me,” she said.

Sally*, in the West Midlands, said: “People are more prone to jump to conclusions about the badge use [now], assuming we’re grifters.”

Sally is a carer for her son, Russell*, 26, who holds a blue badge but can’t drive due to epilepsy, dyspraxia and severe fatigue. Being challenged by the public was a “jeopardy” they always had to keep in mind, she says.

“Just the other day, I was putting the badge into the holder at the front of the car, when this man walked over and stood peering at the window at the badge, not realising it was for my son. It was intimidating.”

These encounters are so distressing for Russell that he often asks to stop their trip out and go home. “My son will say: ‘Oh, Mum, I’m really sorry. It’s all because of me.’ And I say to him: ‘No, it’s because of them.’

“My son looks fit, but he isn’t. Invisible disability is frequently questioned,” she said. “I feel quite worried about his safety, to be honest.”

*Some names have been changed

Additional reporting by Jane Clinton