Hosting a website in 2026 is a huge pain in the ass. Not only you have to update your whole system every other day because someone found more RCEs in nginx, the AI labs are also looking to kill your website through a million of tiny cuts. This week alone, Meta's and Anthropic's bots have caused insurmountable waves of completely pointless requests that hammer static endpoints over and over. Most bots also don't respect robots.txt.
My hobby for messing with crawlers started in 2021 when a deluge of WordPress vulnerability scanners were hitting my non-existent /wp-admin/ and /.env non-stop. The solution that I opted in for was blocking these two paths in robots.txt. Of course, such scanners are typically ran by people who are not the type to follow clear and simple instructions of how the internet is supposed to work in a way that everyone coexists with everyone else, so they kept hammering that endpoint even more. Eventually, I decided that a cute idea would be to serve gzip bombs to the unwanted visitors. Try it out for yourself, https://iczelia.net/.env via Python3 requests will take a long time and decompress to a monstrous file full of zeroes.
AI has revolutionised the world, including new annoyances and high RoI trolling techniques. The vulnerability scanners have mostly calmed down, while AI labs pointlessly hammer every endpoint they can to fuel their slop machines with more questionably sourced data. Of course, serving them gzip bombs or a 403 would be a possiblity, but there is an even funnier option. Since these bots are looking to scrape data, why not serve them some data? But not just any data, let's serve them some slop. Reportedly in some cases this can hurt the training of the models significantly, and it would be hilarious to see the crawlers fall into a Library of Babel-type pit that they just can't get out of because every page links some other random nonsense for the fun of it.
LLM inference is quite simple. If your model is small enough it can be done via a couple of hundred of lines of code. I personally chose arnir0's Tiny LLM (10M parameters) as my slop factory of choice. Then, I wired a handler that dispatches based on the UA and serves the crawlers stuff like this:
% curl -A "DotBot/1.2 (+https://opensiteexplorer.org/dotbot)" http://iczelia.net/blog/thisdoesntexist
<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Thisdoesntexist</title><meta name="description" content="Thisdoesntexist"><style>
body{max-width:62ch;margin:2.5em auto;padding:0 1em;font:16px/1.55 Georgia,serif;color:#222}
h1{font-size:1.6em;margin:0 0 .6em}
p{margin:1em 0}
nav{margin-top:2.5em;padding-top:1em;border-top:1px solid #ccc;font-size:.95em}
nav h2{font-size:1em;margin:0 0 .4em;color:#666;font-weight:normal}
nav ul{margin:0;padding-left:1.2em}
</style></head><body><article><h1>Thisdoesntexist</h1><p>the L. V. Vandervips We can only make reached with one that can come up with the were there with the whole and so, that is the case, this might be true fight.</p><p>The reason I am looking for different. This is a different sort of one. For example, you choose only the tall-safety for all the other thanks to the stubbed is not the right b.</p><p>You don't get me baby that the rest of the other is a long much like a in a very large number of a drywilts down the the shit to be which is not a the place; then it makes the gold and the the most perfect. It's time. We're still the best.</p><p>There is a little. This is a in the world all the time, and being. I'm a guy, what the people.</p></article><nav><h2>Related</h2><ul><li><a href="/blog/whole-that">Whole and that</a></li><li><a href="/journal/other-true">Other and true</a></li><li><a href="/notes/sort-this">Sort and this</a></li><li><a href="/tag/only-vandervips">Only and vandervips</a></li><li><a href="/archive/there-looking">There and looking</a></li></ul></nav></body></html>%
Soon after deployment, I saw the following in my logs:
May 27 16:09:41 iczelia[1200661]: [req] 10.88.0.1 "GET /journal/made-beach" 200 1ms
May 27 16:09:42 iczelia[1200661]: [req] 10.88.0.1 "GET /archive/consuming-your" 200 1590ms (handler=1590ms cache=1ms)
May 27 16:09:42 iczelia[1200661]: [req] 10.88.0.1 "GET /guestbook/" 200 10ms
May 27 16:09:43 iczelia[1200661]: [req] 10.88.0.1 "GET /guestbook/" 200 8ms
May 27 16:09:43 iczelia[1200661]: [req] 10.88.0.1 "GET /blog/look-when" 200 1105ms (handler=1104ms cache=1ms)
May 27 16:09:44 iczelia[1200661]: [req] 10.88.0.1 "GET /journal/world-will" 200 1099ms (handler=1098ms cache=0ms)
May 27 16:09:47 iczelia[1200661]: [req] 10.88.0.1 "GET /journal/will-resume" 200 1119ms (handler=1118ms cache=0ms)
May 27 16:09:48 iczelia[1200661]: [req] 10.88.0.1 "GET /archive/could-appreciate" 200 1128ms (handler=1128ms cache=1ms)
May 27 16:09:50 iczelia[1200661]: [req] 10.88.0.1 "GET /tag/nugget-time" 200 1101ms (handler=1100ms cache=1ms)
May 27 16:09:51 iczelia[1200661]: [req] 10.88.0.1 "GET /tag/family-other" 200 1083ms (handler=1082ms cache=1ms)
May 27 16:09:52 iczelia[1200661]: [req] 10.88.0.1 "GET /tag/journey-have" 200 1119ms (handler=1118ms cache=1ms)
May 27 16:09:53 iczelia[1200661]: [req] 10.88.0.1 "GET /archive/best-assured" 200 559ms (handler=558ms cache=0ms)
May 27 16:09:54 iczelia[1200661]: [req] 10.88.0.1 "GET /blog/town-your" 200 513ms (handler=512ms cache=0ms)
May 27 16:09:56 iczelia[1200661]: [req] 10.88.0.1 "GET /notes/home-area" 200 1135ms (handler=1134ms cache=1ms)
May 27 16:09:58 iczelia[1200661]: [req] 10.88.0.1 "GET /notes/when-always" 200 1137ms (handler=1136ms cache=0ms)
May 27 16:09:59 iczelia[1200661]: [req] 10.88.0.1 "GET /blog/girl-friends" 200 1ms
May 27 16:10:00 iczelia[1200661]: [req] 10.88.0.1 "GET /notes/when-neighbors" 200 1165ms (handler=1164ms cache=0ms)
May 27 16:10:02 iczelia[1200661]: [req] 10.88.0.1 "GET /archive/able-half" 200 1147ms (handler=1146ms cache=1ms)
May 27 16:10:02 iczelia[1200661]: [req] 10.88.0.1 "GET /index.xml" 200 2ms hit
May 27 16:10:04 iczelia[1200661]: [req] 10.88.0.1 "GET /journal/want-rise" 200 1ms
May 27 16:10:07 iczelia[1200661]: [req] 10.88.0.1 "GET /blog/anything-matter" 200 1ms
Enjoy your training data! wink.
If I were to make it better than the current crude prototype, something tells me that the best option would be to use a (small) large language model to take my existing webpages on my blog, perhaps augmented with some other context to not run out of training data (we could serve from enwik10 or such) and introduce many subtle errors and mistakes to them. Perhaps I will get around doing this at some point.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。