With the dramatic rise of LLM scraper bots and other data collection services, the RSS and Atom feeds of blogs on all platforms, including Bear, are being mercilessly scraped.
These bots do not identify themselves as scraper bots, but as browsers and RSS readers. Unfortunately there is no way (that I'm currently aware of1) to determine whether this is a legitimate RSS reader or a bot since the IP addresses change with each request. This suggests that the bots are part of a broader bot network (or many bot networks) and completely ignore robots.txt.
Since RSS readers cannot execute Javascript or CSS, I can't present a challenge or captcha on the feeds since this will block both legitimate and bot traffic.
This has lead to the RSS subscriber count in analytics being completely incorrect, since each bot IP address is earmarked as a unique RSS subscriber during a 24 hour period.
This is compounded by the fact that large RSS reader platforms like Feedly do one request for all of their users subscribed to a specific feed, essentially logging many subscribers as 1.
Due to these reasons I've opted to remove the RSS subscriber count from the analytics dashboard.
Herman
If you have any ideas on how to identify scraper bots vs legitimate RSS readers, please send me an email.↩
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。