Systemand method for enabling website owners to manage crawl rate in a website indexing system
US8032518B2 Патенты
website for purposes of indexing the documents for search engines. The web crawlers crawl a specified website at a crawl rate that is based on multiple factors. One of the factors is a pre-set crawl rate limit. According to certain embodi ments, an owner for a specified website is enabled to modify the crawl rate limit for the specified website when one or more pre-set criteria are met.
Притязаний: 36. Знаков текста: 110 323.