惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

爱范儿
爱范儿
Simon Willison's Weblog
Simon Willison's Weblog
K
Kaspersky official blog
P
Palo Alto Networks Blog
Google DeepMind News
Google DeepMind News
www.infosecurity-magazine.com
www.infosecurity-magazine.com
AI
AI
G
GRAHAM CLULEY
O
OpenAI News
Application and Cybersecurity Blog
Application and Cybersecurity Blog
Help Net Security
Help Net Security
L
LINUX DO - 热门话题
S
Schneier on Security
P
Privacy International News Feed
L
Lohrmann on Cybersecurity
SecWiki News
SecWiki News
C
Cybersecurity and Infrastructure Security Agency CISA
T
Threatpost
C
Cyber Attacks, Cyber Crime and Cyber Security
A
Arctic Wolf
C
Cisco Blogs
V2EX - 技术
V2EX - 技术
有赞技术团队
有赞技术团队
Apple Machine Learning Research
Apple Machine Learning Research
月光博客
月光博客
Latest news
Latest news
人人都是产品经理
人人都是产品经理
Schneier on Security
Schneier on Security
Last Week in AI
Last Week in AI
Webroot Blog
Webroot Blog
美团技术团队
N
News and Events Feed by Topic
大猫的无限游戏
大猫的无限游戏
Security Archives - TechRepublic
Security Archives - TechRepublic
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
博客园 - 三生石上(FineUI控件)
The Cloudflare Blog
Project Zero
Project Zero
博客园_首页
Cloudbric
Cloudbric
IT之家
IT之家
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
雷峰网
雷峰网
罗磊的独立博客
Hacker News: Ask HN
Hacker News: Ask HN
The Last Watchdog
The Last Watchdog
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
T
Tenable Blog
Scott Helme
Scott Helme

Giant Robots Smashing Into Other Giant Robots

Join us: Building Secure Healthcare Systems Upcase has retired, but the learning continues The Bike Shed Ep 506: The Muppet Software Team Migrating to native stack navigation, with a surprise from iOS 26 Past and present thoughtbotters at LRUG this Monday The Bike Shed Ep 505: What is a “principal” or “staff” engineer? Your vibe coded website is going to get you fined Roux’s New Component Library Why we're choosing stewardship over an exit The Bike Shed Ep 503: Seeing the Graph for the Trees Announcing Shoulda Matchers 8.0: validate multiple attributes in one line AI's "overnight" solution for our flaky tests took two weeks to adopt The Playwright debugging tool Rails devs aren't using Meet thoughtbot at Brighton Ruby 2026 614: AI Code Audits The mistake I didn't realise I was making when designing workshops 502: Apps That Make Our Work Go Toast: the 2-minute test that reveals how you think about building products How I Built a Chrome Extension Wrapper (and Everything That Tried to Stop Me) Enforcing Your Ruby Style Guide on AI-Generated Code Copy as Markdown: AI-friendly blog posts 501: What makes for good technical writing? The Four Signals of AI Observability Can you really launch a tech business with a no-code app builder? 612: Do fish drink? This week in #dev (May 15, 2026) Lost, forgotten, and unfamiliar HTML 500: Celebrating with past hosts Why Duck Typer? Biometrics authentication for your mobile app
AI crawlers are inflating your view counts
Trésor Bireke · 2026-06-16 · via Giant Robots Smashing Into Other Giant Robots

Your most-viewed page might be one no human has ever opened. That is what AI crawlers have done to view tracking in 2026.

I ran into this problem on a production app that needed engagement tracking. The first version tracked everything server-side, the way Rails apps have done analytics for years. It broke within a day.

The problem: crawlers inflate every count

We used Ahoy for tracking. Each controller action called ahoy.track while rendering the page, and every event rolled up into a denormalized counter column with counter_culture.

The issue is that server-side tracking fires on every request, including bots. AI crawlers like Meta-ExternalAgent, Bytespider, and Baiduspider were making roughly 100,000 requests per day. They were not attacking the site, just reading to feed training pipelines.

Ahoy has bot detection built in. It uses the device_detector gem to check user agents and skips known bots. That list catches Googlebot and older crawlers, but it misses the new wave of AI crawlers. As a result, every one of those requests created an Ahoy::Event row and incremented the corresponding counters.

Our view counts were not measuring human interest. They were measuring how hungry the scrapers were that week.

Fix one: require JavaScript

Chasing user agent strings is a losing game. New crawlers appear faster than blocklists update. But there is one thing AI crawlers reliably do not do, and that is execute JavaScript.

So we moved view tracking out of the controllers. Pages declare what is trackable as a data attribute, and a small Stimulus controller fires a beacon after the page loads.

connect() {
  if (this.element.dataset.viewTrackerFired === "true") return
  this.element.dataset.viewTrackerFired = "true"

  const fire = () => this.fire()
  if ("requestIdleCallback" in window) {
    requestIdleCallback(fire, { timeout: 2000 })
  } else {
    setTimeout(fire, 500)
  }
}

A few details mattered here:

  • requestIdleCallback defers the beacon until the browser is idle, so tracking never competes with rendering. The 2-second timeout guarantees it still fires on busy pages.
  • keepalive: true on the fetch lets the request survive the user navigating away immediately.
  • The fired flag guards against Turbo reconnecting the controller and double-counting.

Crawlers fetch the HTML and move on. Real browsers run the beacon and get counted. View counts dropped sharply the day this deployed. That was the fix landing, not a regression.

Fix two: the bots found the beacon

Three days later, the tracking endpoint /track/events was the most-crawled path on the site. Crawlers do not execute JavaScript, but they do parse it. The endpoint URL sits in the markup as a data attribute, so the scrapers extracted it and started requesting it directly.

None of those requests created events, but they still burned through the full Rails stack for nothing. The fix was two cheap layers.

First, robots.txt for the well-behaved bots:

Second, a guard in the controller for everyone else:

class TrackingEventsController < ApplicationController
  before_action :reject_bots

  private

  def reject_bots
    head :no_content if DeviceDetector.new(request.user_agent).bot?
  end
end

Any request with a bot user agent gets a 204 before the action runs. No parsing, no resource lookups, no database work. The well-behaved crawlers respect robots.txt and never arrive, and the rest get the cheapest possible response.

The takeaway

Server-side analytics was built for a web that no longer exists. In 2026, a meaningful share of your traffic comes from AI crawlers, so counting views on the server measures scraper appetite, not audience.

The defense is not one clever trick. It is stacked cheap layers: robots.txt for the bots that ask permission, a user agent check that returns early for the ones that announce themselves, and a JavaScript beacon for the bots that do neither.

Check your own numbers. If your view counts have never had a suspicious cliff in them, the bot tax is probably still baked in.