at war with the machines

The easiest solution for these scrapers is to refuse to serve them. Identify them via well known User-Agent or IP range, and reject their requests. This works briefly, until they realise they are being stonewalled and opt to come back on residential IPs with masked user agents. When you refuse, they adapt. ^[1]

Similarly, serving static content when a scraper is detected yields the same result, as they realise every path leads to the same thing, which is very atypical for a normal website.

We’ll need something smarter to combat these scrapers, so what’s out there? There are two emerging strategies, that of Anubis (proof of work), and that of Iocaine (tarpitting).

Anubis focuses on making browsing expensive, by placing a JavaScript challenge in front of a resource that requires nontrivial computation to solve. This should have minimal impact on real users, but places financial burden on scrapers who request many many more pages from distinct agents.

Unfortunately, Anubis has several shortcomings.

With all of these shortcomings in mind, how does Iocaine stack up?

Iocaine proxies all of your incoming requests ^[3], detects scrapers, and sends them into an infinite maze if it finds one. The exact method of detection can be configured by you, with the provided configuration checking against a list of known bad user agents ^[4].

The content that Iocaine serves can again be configured by you, but the general principle remains the same. It will take a starting corpus and generate garbage but plausible text using a markov chain. This means that scrapers will get unique, non static text every time, without causing significant load on your infrastructure.

Importantly, though, Iocaine will serve the same text for the same URL, this stops scrapers recognising they are getting completely random data. It is recommended that you rotate the Iocaine RNG seed every so often in order to make it look like your content changed naturally, which keeps the scrapers interested.

Iocaine will include many links in its default template, which all point deeper into the maze. Scrapers can explore around free as a bird, but there is no escape. You can see Iocaine in action by visiting the demo page at https://poison.madhouse-project.org/

One key advantage of Iocaine is that it inverts the model of detection. Once a scraper enters the pit, it will remain there forever. Iocaine picks up on the fact that the URL being requested already points to the maze and immediately serves garbage, meaning that for any link in the chain, the entire chain is poisoned. Even if the scraper comes back later, they will still get garbage, every single time.

This is in contrast to Anubis, which requires you to constantly prove you are not a bot.

The primary flaw with Iocaine is that it relies on the initial detection in order to be effective. If scrapers adapt and start masking, Iocaine will become helpless.

It may have been clear from my description of each approach that I fastly prefer Iocaine. Admittedly I did not try Anubis, though this was simply because I don’t agree with slowing everyone down for little gain.

I followed the basic Iocaine deployment guide ^[5], including the NGINX guide for its integration there, and that was it. The traffic reaching my underlying infrastructure disappeared with the flick of a switch.

Iocaine is deployed on a cheap VPS, which I was already using to give me a static IP. This means I don’t even serve the garbage from my own uplink, further reducing bandwidth used.

Now that we are protected, let’s set up some monitoring to see what the scrapers are getting up to. Iocaine exposes prometheus metrics for us to hook into, which in combination with NGINX log monitoring lets us see exactly what’s happening.

I chose VictoriaMetrics & VictoriaLogs for my storage solution, as they are well suited to run with minimal resource. I had tried SigNoz but it requires 16+GB of RAM to even function! Grafana was used as my visualiser, being the de-facto implmentation with great support for the Victoria stack.

Now, let’s dig deeper.

At the time of writing, I have the last 7 days of data from the stack as per the VictoriaLogs default retention policy. It is a sane default for Victoria to only retain 7 days of data, as it currently retains around 1M logs/day.

There was interesting activity which I captured before then, though I am relying on posted screenshots.

Anthropic decided it really wanted to know what was in my maze, so sent a small army to investigate. The scale of this abuse is just insane. If I was serving these requests from my infrastructure, it would’ve almost certainly taken it down.

Not shown here is Anthropic previously taking little peeks into the maze, sending the odd request to see what' around. To me, it looks like they were scoping things out before going to town.

This is a typical week for me, steady requests coming from the scrapers. What I do find interesting here is that OpenAI slowed way down after my NGINX proxy went off for a few minutes. This wasn’t caused by them, it was me doing work on it, but still it looks like they back off in response to the timeouts.

These are Iocaine’s metrics, visualised in various ways. The 116GB figure is the alltime generation, since I first deployed it, as is the distribution of garbage. It is obvious that the scrapers want to steal the code from my forge, which they know exists as they have seen Forgejo on that subdomain previously.

The garbage vs human metric is represented in requests per second. You can see that nearly all of the traffic is garbage.

As I’ve been collecting data on what my proxy sees, it’s made me realise just how hostile the internet is. No longer is there a subtle background hum, the odd request for /.env. Nowadays, I see regular fully fledged attacks on my infrastructure, exploiting various 0days and persistently trying to read all fashion of secret files from every domain I host.

On top of the well known LLM scrapers, I’ve seen several smaller operations scraping data from my homelab. A few highlights:

Most are benign, though some are abusive. I have done the work to combat those, but that will have to wait for another time…​