Distributed crawling of
US8812478B1 Patents
Techniques for crawling hyperlinked documents are pro vided. Hyperlinked documents to be crawled are grouped by host and the host to be crawled next is selected according to a stall time of the host. The stall time can indicate the earliest time that the host should be crawled and the stall times can be
17 claims. 39,985 characters of text.