User-agent: *
Disallow: /
now i some middleware that looks at the header, and if they are polite enough to include "bot" in the user agent, they politely get a 404 response.
now i some middleware that looks at the header, and if they are polite enough to include "bot" in the user agent, they politely get a 404 response.
#AI #Liability
https://www.euractiv.com/section/tech/news/commission-withdraws-ai-liability-directive-after-vance-attack-on-regulation/
#AI #Liability
https://www.euractiv.com/section/tech/news/commission-withdraws-ai-liability-directive-after-vance-attack-on-regulation/
User-agent: *
Disallow: /
seems to work. Or maybe those bastards change their user agent and claim to be someone nice. In any case, I just added a bunch of
location = /robots.txt {
add_header Content-Type text/plain;
return 200 "User-agent: *\\nDisallow: /\\n";
}
in my nginx config. No need for any bot to visit, crawl and index most of my sites.=
User-agent: *
Disallow: /
seems to work. Or maybe those bastards change their user agent and claim to be someone nice. In any case, I just added a bunch of
location = /robots.txt {
add_header Content-Type text/plain;
return 200 "User-agent: *\nDisallow: /\n";
}
in my nginx config. No need for any bot to visit, crawl and index most of my sites.=
robots.txt file. only noticed it because the OpenAI bot was hitting me with a lot of nonsensical requests. here is the list from last month:- (810) bingbot
- (641) Googlebot
- (624) http://www.google.com/bot.html
- (545) DotBot
- (290) GPTBot
- (106) SemrushBot
- (84) AhrefsBot
- (62) MJ12bot
- (60) BLEXBot
- (55) wpbot
- (37) Amazonbot
- (28) YandexBot
- (22) ClaudeBot
- (19) AwarioBot
- (14) https://domainsbot.com/pandalytics
- (9) https://serpstatbot.com
- (6) t3versionsBot
- (6) archive.org_bot
- (6) Applebot
- (5) http://search.msn.com/msnbot.htm
- (4) http://www.googlebot.com/bot.html
- (4) Googlebot-Mobile
- (4) DuckDuckGo-Favicons-Bot
- (3) https://turnitin.com/robot/crawlerinfo.html
- (3) YandexNews
- (3) ImagesiftBot
- (2) Qwantify-prod
- (1) http://www.google.com/adsbot.html
- (1) http://gais.cs.ccu.edu.tw/robot.php
- (1) YaK
- (1) WBSearchBot
- (1) DataForSeoBot
i have placed some middleware to reject these for now but it is not a full proof solution.
robots.txt file. only noticed it because the OpenAI bot was hitting me with a lot of nonsensical requests. here is the list from last month:- (810) bingbot
- (641) Googlebot
- (624) http://www.google.com/bot.html
- (545) DotBot
- (290) GPTBot
- (106) SemrushBot
- (84) AhrefsBot
- (62) MJ12bot
- (60) BLEXBot
- (55) wpbot
- (37) Amazonbot
- (28) YandexBot
- (22) ClaudeBot
- (19) AwarioBot
- (14) https://domainsbot.com/pandalytics
- (9) https://serpstatbot.com
- (6) t3versionsBot
- (6) archive.org_bot
- (6) Applebot
- (5) http://search.msn.com/msnbot.htm
- (4) http://www.googlebot.com/bot.html
- (4) Googlebot-Mobile
- (4) DuckDuckGo-Favicons-Bot
- (3) https://turnitin.com/robot/crawlerinfo.html
- (3) YandexNews
- (3) ImagesiftBot
- (2) Qwantify-prod
- (1) http://www.google.com/adsbot.html
- (1) http://gais.cs.ccu.edu.tw/robot.php
- (1) YaK
- (1) WBSearchBot
- (1) DataForSeoBot
i have placed some middleware to reject these for now but it is not a full proof solution.
robots.txt file. only noticed it because the OpenAI bot was hitting me with a lot of nonsensical requests. here is the list from last month:- (810) bingbot
- (641) Googlebot
- (624) http://www.google.com/bot.html
- (545) DotBot
- (290) GPTBot
- (106) SemrushBot
- (84) AhrefsBot
- (62) MJ12bot
- (60) BLEXBot
- (55) wpbot
- (37) Amazonbot
- (28) YandexBot
- (22) ClaudeBot
- (19) AwarioBot
- (14) https://domainsbot.com/pandalytics
- (9) https://serpstatbot.com
- (6) t3versionsBot
- (6) archive.org_bot
- (6) Applebot
- (5) http://search.msn.com/msnbot.htm
- (4) http://www.googlebot.com/bot.html
- (4) Googlebot-Mobile
- (4) DuckDuckGo-Favicons-Bot
- (3) https://turnitin.com/robot/crawlerinfo.html
- (3) YandexNews
- (3) ImagesiftBot
- (2) Qwantify-prod
- (1) http://www.google.com/adsbot.html
- (1) http://gais.cs.ccu.edu.tw/robot.php
- (1) YaK
- (1) WBSearchBot
- (1) DataForSeoBot
i have placed some middleware to reject these for now but it is not a full proof solution.
slept like shit. fought back the urge to shit for four fifths of the run. bleh.
#running
slept like shit. fought back the urge to shit for four fifths of the run. bleh.
#running
slept like shit. fought back the urge to shit for four fifths of the run. bleh.
#running
(At first, I thought the touchpad of that laptop was broken, because a right click on the desktop didn’t do anything. But it worked just fine. It just takes ~10 seconds for the popup to show.)~
(At first, I thought the touchpad of that laptop was broken, because a right click on the desktop didn’t do anything. But it worked just fine. It just takes ~10 seconds for the popup to show.)~
(At first, I thought the touchpad of that laptop was broken, because a right click on the desktop didn’t do anything. But it worked just fine. It just takes ~10 seconds for the popup to show.)~
(At first, I thought the touchpad of that laptop was broken, because a right click on the desktop didn’t do anything. But it worked just fine. It just takes ~10 seconds for the popup to show.)~
#running #treadmill
#running #treadmill
#running #treadmill
legs are soooo tight. felt light though.
#running
legs are soooo tight. felt light though.
#running
legs are soooo tight. felt light though.
#running
#ilovefs
https://ansol.org/eventos/2025-02-14-ilovefs/