News.com writes about LookSmart’s acquisition of Grub in the hope that “distributed computing [will] improve its Web search results.” This is interesting because perhaps silimar ideas could be applied to weblogs.
“Most engines only update their entire document catalog once a month, because there’s an inherent computing problem: They can’t do it any faster,” said Pete Adams, chief technology officer of LookSmart. “The goal of this technology is to be able to crawl every document on the Internet every day. We can only do that if we can grow the number of people that are running the software–the computing power we would use is a function of how many people we have donating their computer power.”
Danny Sullivan, editor of the industry newsletter Search Engine Watch, speculated that LookSmart might use the Grub system to start a “trusted feed” service for inclusion into its WiseNut index. Marketers could send updated Web pages to the index to refresh it for a fee–or what’s known as paid inclusion. Search engines, including Inktomi and Fast’s AlltheWeb, already use such a service to keep indexes of product-related sites and catalogs fresh and to augment revenue.
