Retrospective vs Prospective Search

Bob Wyman of PubSub had a post in April which discusses two types of search:

  • Retrospective Search: That which is done by traditional search engines — “Searching the past”. This kind of search typically relies on net crawlers, spiders or other data entry methods to gather files into a historical, searchable collection. While the collection changes over time, at the time of any specific “search”, the collection can be considered to be static. Any single search query will only be evaluated against those documents that were collected prior to the moment the query was submitted to the search engine. On the other hand, the queries in such a system are constantly changing and are thus dynamic.

  • Prospective Search: What PubSub does — “Searching the future”. This kind of search can depend on a wide variety of means to gather newly updated documents but does not rely on a static, historical collection. In a prospective search, a query is registered in a collection of queries and then evaluated against every new document as it is discovered. Thus, while the query is static, the collection of documents against which it is evaluated is dynamic. Also, the entire collection of currently active queries can be considered “static” at the moment that any new document arrives in the system.

    So, a retrospective system is characterized by a static document collection and dynamic, single-use queries while a prospective search system will have, at any moment, a static set of persistent queries and a dynamic document under evaluation. Retrospective systems typically need to store large numbers of documents but have no need to store queries. Prospective systems need to store queries but can discard documents as soon as the collection of persistent queries has been evaluated against them.

    These two styles of “search” address both different and complimentary needs. They are often both required as part of a comprehensive program for discovering and keeping knowledge up-to-date. For instance, it is often useful to first do a retrospective search to discover “what is known” about an area of interest and then to establish a persistent, prospective search in order to be notified “whenever something new appears.” (You might, for instance, search the archives of the New York Times for stories about “copyright law” and then create a “news alert” to be notified whenever they write a new story on the subject.)

  • Published by

    Rajesh Jain

    An Entrepreneur based in Mumbai, India.