robots文件检测工具

百度搜索资源平台：https://ziyuan.baidu.com/robots/
查看搜索引擎爬虫访问记录

[root@]# cat access.log|grep Baiduspider|wc -l
596650
[root@]# cat access.log|grep Googlebot|wc -l
540340
[root@]# cat access.log|grep 360Spider|wc -l
63040
[root@]# cat access.logt|grep bingbot|wc -l 
621670
[root@]# cat access.log|grep YisouSpider|wc -l 
3800100
[root@]# cat access.log|grep Sogou|wc -l
533810