The limitations of a crawl and how to overcome them