nginx 配置拒绝爬虫,收录
作者：互联网
在nginx的http段中加入如下配置,达到拒绝爬虫,收录的效果,返回403无权限信息
if ($http_user_agent ~* "ab|wget|MJ12bot|qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|compatible|YandexBot|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo ! Slurp China|YoudaoBot|compatible|Sosospider|Sogouspider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot|YodaoBot|Bingbot|Teoma|twiceler|Scrubby|Robozilla|Gigabot|yahoo-mmcrawler|yahoo-blogs/v3.9|psbot") {
return 403;
}
配置前后可在站长工具中实验收录查询的效果标签：Google,http,Googlebot,Yahoo,爬虫,nginx,403,收录 来源： https://blog.51cto.com/zhangxiaoxiong/2428960