nginx配置禁止爬虫配置
作者：互联网
1、在配置文件里添加禁止爬虫配置
server {
listen 80;
server_name 127.0.0.1;
#添加如下内容即可防止爬虫
if ($http_user_agent ~* "qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot")
{
return 403;
}
标签：Google,Googlebot,配置,spider,爬虫,nginx,Yahoo,server 来源： https://www.cnblogs.com/zhj5418/p/14517033.html