Single post
jump to replies
Meta's bot crawler is really stupid.
Although I blocked its user agent yesterday, returning a 403 Forbidden reply, it keeps on crawling my online copy of wikipedia 🤪
grep "meta-externalagent" access.log | grep " 403 " | wc -l
3816
3 replies
back to top
@adele a crawler should honor rules in robots.txt - but they don’t have to. Their bot was probably told to ignore certain status codes. perhaps give it a 404 instead. If that doesn’t work, Try a 3xx code like a 301 ;)
@awoodsnet 301, why not, but towards which page ??
@adele if you think about it like a lost puppy, you could send it home … to meta.com 😂
But i’d start with a 404 to see if that does the trick