My latest Business Standard column:
Search has become a window to the world wide web of data. Todays search is simplistic: type a few words in a box, get back zillions of results, and click on one or more of the results to see if we get what we are looking for. Think of todays search as the DOS era: a good start, but not enough to unleash the real power of what can be. It took a decade to go from DOS to Windows. It has taken us almost as long to start imagining and working towards the next generation of search technologies. Here are some key ideas which will help define tomorrows search:
Integration between Desktop and Internet Search: One of the great inconsistencies of our computing world has been the disconnect between being able to search the web much more easily than our own desktop computer. This is beginning to change with the emergence of desktop search utilities like Copernic, Blinkx and Hotbots desktop search toolbar. What is still missing is seamless integration between our personal data (spread across emails, attachments, IM logs and documents) and the web.
Better Visualisation and Navigation Tools: Is a set of pages, each with 10-20 results, the best way to navigate the millions of search results that can show up? It is only now that ideas from information visualisation (for example, Groxis) are making their way for viewing of search results. By clustering results and providing visualisation techniques, it should be possible to provide rich interfaces for navigation.
Real-time Search: The current model involve search engines crawling web pages every so often and including them in their search results. This means that at times it can be several days before new updates on a page can show up in search engine results. There are two ways to address this problem: websites can notify search engines via a ping when they are updated (much like weblogs do to sites like weblogs.com), and search engines can dynamically query specific databases via web services APIs to deliver results (much like Googles own API offers). A third approach has emerged with narrowly focused RSS- and XML-based search engines like Feedster, Technorati and PubSub.com.
Searchstreams Analysis: What searchstreams offer, according to John Battelle of Searchblog, is the ability to capture and record your search history as well as the things you looked at, all in one package. Battelle, who is also writing a book on search, explains further: What I really wish for, both to tell the story of my search, and to annotate my book, is the ability to take that searchstream and turn it into an object – a narrative thread of sorts, something I can hold and keep and refer to, a prop to aid in the telling and retelling of how I came to my answer. Tracks in the dust, so to speak, so others can follow and make their own, or follow mine and see (and question!) how I came to my conclusions. Imagine, I thought to myself, if instead of footnotes and citations, I could append searchstreams… Next-generation infoware tools like Furl (recently acquired by Looksmart) and del.icio.us are already working to enable some of this. Searchstream analysis could also be used to add context and personalisation to search. Searchstreams also offer a foundation to construct Vannevar Bushs vision of the Memex.
Multimedia Search: Our world of data has grown to beyond search. Even though there is image search available on search engines, it is restricted to the text and keywords associated with the images. What is needed is the ability to look inside sound files, images and videos and allow search based on the content. This becomes especially important in the context of the growing ease of generating and publishing multimedia content via cellphones.
Mobile Devices: Talking of cellphones, it is going to become increasingly important to support a quality search experience on these devices. The current search results are best shown on a web browser. Yet, in countries like India, the number of mobile devices computers. So, the focus for search needs to be on providing accurate results in the form of microcontent that can be sent and displayed on mobile devices. Not only does this mean a different format, but it could also mean providing location-based information.
Local Search: Much of our life is spent in neighbourhoods. It is quite hard to find local information in the vicinity of where we live and work. It is estimated that a quarter of the searches performed on search engines have a local flavour to them. Being able to integrate the results with maps, directions and other neighbourhood-specific information can make search engines much more relevant. This is where search engines like Yahoo and Google will compete head-on with classifieds and yellow pages.
Vertical Search: Chris Sherman wrote on SearchEngineWatch that searchers are becoming more sophisticated, and are learning that general purpose search engines are not always the best choice for every type of search. Because these search engines have a narrower focus and limited domain, they can offer specialized utilities for a richer experience. An example of vertical search engines is GlobalSpec, which focuses on the engineering industry.
The state of Search is very much like the way the scientific world was in the seventeenth century until Issac Newton came along and helped lay the foundation for the world ahead with his theories and inventions. A similar revolution is needed in the world of Search. Can we in India play a role, just as we did in some of the mathematical discoveries many centuries ago?