US20130117252A1
патент
System and method for fetching embedded object content as part of a batch crawl. A fetch server receives a request on a request thread to retrieve content for objects embedded in a document, such as a web page. The fetch server attempts to locate the content of the object in cache first and in disk storage next. If the content is not located in the cache the fetch server may switch the request to a worker thread. If the content is not located in the disk storage, the fetch server may schedule a request to retrieve the content of the embedded object through a batch web crawl. Scheduling a request may include determining that a request to crawl the content of the object has already been scheduled or inserting a request into a scheduling queue.
Притязаний: 20. Знаков текста: 48 807.