|
如题,按照下面两种方式设置,搜索引擎的爬虫还是会被屏蔽掉
1. valid_referers *.aaa.com server_names ~\.google\. ~\.baidu\.;
2. valid_referers *.aaaa.com *.google.com *.baidu.com;
123.125.71.105 - - [17/Jun/2017:00:45:51 +0800] "GET /******** HTTP/1.1" 403 162 "-" "Baiduspider-image+(+http://www.baidu.com/search/spider.htm)" - |
|