Hosting public site, you’ve dealt with them already.
\nUsing your favorite search engine, you’re indirectly subject to their work as well.
\nCrawlers are bots, querying and sometimes mapping your site, completing some search engine database.<\/p>\n
When I started looking at the subject, a few years back, you only had to know about \/robots.txt, to potentially\u00a0prevent your site from being indexed, or at least restrict such accesses to relevant contents\u00a0of your site. At this point, indexing scripts and style-sheets, you might say – I’m surprised not to find the remark in the comments – that google actually indexes your site vulnerabilities, creating not only a directory of the known internet, but a complete map with everyones’\u00a0hidden pipes, that could some day be used to breach your site – if not already.<\/p>\n Even if Google is the major actor on that matter, you probably have dealt with Yahoo, Yandex.ru, MJ12, Voltron, … which practices are similar.\u00a0Over the last years, checking your web server logs, you might have noticed a significant increase in the proportion of bot queries over human visits. In part due to search engines recrudescence, though I suspect mostly thanks to bot nets. Assuming your sites are subject to DDOS attacks, scans for some software\u00a0vulnerability (top targeted solutions being wordpress, phpbb\u00a0and phpmyadmin), you should know attackers will eventually masquerade their user-agent. Most\u00a0likely branding themselves as Googlebot.<\/p>\n To guarantee\u00a0a “googlebot” branded query actually comes out\u00a0some google server, you just need to check the pointer record associated to this client’s IP. A way to do so in Apache (2.4) could be to use something like this<\/a>\u00a0(PoC to complete\/experiment).
\nMore recently, we’ve seen the introduction of some XML files such as sitemap, allowing to efficiently serve to search engines a map of your site.
\nThis year, Google reviewed his “rules”<\/a> to prioritize responsive and platform-optimized sites as well. As such, they are now recommending to allow crawling for JavaScript and CSS\u00a0files, warning –threatening<\/a>– that preventing these accesses could result in your ranking being lowered.<\/p>\n
\nIdentifying these crawlers could be done checking the UserAgent, sent with\u00a0all http requests. On small traffic sites,\u00a0crawlers may very well be your only clients.<\/p>\n
\nStill, maybe is it wiser to just drop all google requests as well. Without encouraging Tor usage,\u00a0it’s probably time to switch to DuckDuckGo?<\/p>\n