social.pollux.casa

You don't know what #religion I am?

That's normal, I keep that to myself. Maybe it's because I'm French and in my country secularism is a dogma. For me, religion is part of intimacy and privacy.

Block non-human crawlers with lighttpd

2025-04-20 19:05

Recently, I've put a copy of some ZIM files online with kiwix-server. I posted the url of this site on the Fediverse and, a few days later, the little server was a bit overloaded. The logs showed that the site was being crawled by search engines and AI training bots. There was no reason to let them. A robots.txt file calmed some, but not others.

Analysing user agents and IP addresses is not the answer, because, everything is done to make it complicated (randomisation, many datacenter origins). I thought about Cloudflare protection, Google captcha or the open source solution Anubis. All of them require javascript to be enabled on the human browsers.

After several tests, I have found a simple method to stop these crawlers.

The principle

When a connection arrives on the web server, it checks to see if the request comes with a cookie. If it does not, the web server redirects the browser to an HTML form that asks the user to tick a checkbox and submit. If the user submits the form correctly, he or she receives a cookie and is redirected to the previously requested page. The new request is made with a cookie. So, the web server does its job and send the expected content.

Detail on my blog

I think I've managed to curb AI crawlers with a simple HTML form that drops a cookie.

I'll see if it holds up and make a post about it.

It protect my #kiwix server

https://zim.pollux.casa/

w is the preferred subdomain for smolweb sites

smolweb news

A simple way for a website to indicate that it adheres to the smolweb guidelines is to use the subdomain w instead of www.

This short subdomain is very much in the spirit of smolweb.

This is not a requirement, just a suggestion.

w.example.org is visually different from www.example.org or example.org.

#smolweb #smallweb

New smolweb symbol could be <w> in monospaced font

The angle brackets < and > recall HTML tags, giving a nod to the DIY, hand-coded nature of many smolweb sites. It signals that the web can be understandable and writeable by humans, not just something consumed through complex platforms. It carries the scent of rawness and control, not abstraction and obfuscation.

A single, lowercase w stands in for "web" in a modest, almost shy way. It’s not the grand "www" of the mainstream web, but a small w, lowercase and self-aware. It hints at an alternative web, not the bloated, surveillance-driven one, but a quieter, slower, more intimate one.

<w> is compact, echoing the idea that less is more. That brevity is not a compromise but a strength. It suggests that meaning and value can come in small, lightweight package, just like a well-crafted static HTML page.

The whole symbol feels like something you might type into a terminal or a text editor. It invites tinkering. It feels like the smolweb: accessible, understandable, and remixable.

So yeah, <w> quietly says a lot: it’s webby, small, and stripped-down, no frills, all signal. Perfect shorthand for the smolweb !

https://smolweb.org/

If you run a single user #GoToSocial instance, maybe you would like to get more toots about a specific subject in your timeline.
#FediBuzz relay could help you. You can subscribe to a specific hashtag fake-account and then receive toots it re-toots.

more info here https://relay.fedi.buzz/

For example, you can follow @tag-gotosocial