Jeremy Zawodny writes that “today’s leading search engines weren’t designed to work in a world with millions of feeds.
In the old web, most content never changes. The only exceptions seem to be traditional news outlets (CNN, New York Times, and so on). That means search engines can crawl most sites infrequently and nobody really notices. Missing the last few days worth of stuff isn’t that big a deal as long as you crawl those “news” sites regularly. Also, there’s no way to find out which pages have changed without crawling the whole site, and that’s quite expensive.
In the new world, feeds update frequently. Blogs start to look a like like “news” sites to search engine crawlers. But the updates are contained within the feed, so there’s no need to crawl every link on the site looking for the new stuff. In other words, the cost of staying current on site changes is much lower when feeds are available.
Once this feed stuff hits the tipping point (I think we’re close), things will get really, really interesting. Suddenly these feed sources will be the thing people care about. The model of “search and find” or “browse and read” will turn into “search, find, and subscribe” for a growing segment of Internet users and it will really change how they deal with information on the web.
Rich Skrenta calls it “The Daily Internet.”
The proliferation of incremental content sources, all pumping out new material on a regular basis, is what the mainstream Internet user will consume. It’s the difference between doing research or reading a magazine. At Topix.net we believe that editorial automation is necessary to manage this massive, growing content stream. Other startups like Feedster and Technorati are also focused on improving access to the incremental Internet. This is the future of audience on the net, as well as the next online advertising frontier.
John Battelle adds: “Content is king, search is the king’s phonebook (or index…). I certainly agree, but the two are entirely intertwingled. Google et al do well when content does well, otherwise, what is there to index? The big question, however, is whether Google wants to start playing in the content aggregation space – the new model of ‘search, find, subscribe’ that Jeremy discusses. Yahoo has already decided it does (and the buzz is that more is coming…). So has MSN and AOL. But at the end of the day, the tail is far too big on this beast, and no one place can own the conversation.”
