Thread of 2 posts
Meta's bot crawler is really stupid.
Although I blocked its user agent yesterday, returning a 403 Forbidden reply, it keeps on crawling my online copy of wikipedia 🤪
grep "meta-externalagent" access.log | grep " 403 " | wc -l
3816
Claude's bot is worth!
grep "ClaudeBot" access.log | grep " 403 " | wc -l
30076