Tag: Crawler

Crawlers Bias Server Log Stats

Search engines are regularly visiting web sites to update their index. The shorter the time between two visits the more accurate is the index. Of course, there are side effects. On one hand this increases the load and the traffic of a web server. Thus, most modern crawlers are not too aggressive to avoid too …

Continue reading