TECH TALK: The Future of Search: Simple Silly Search

The hottest topic in computing and Internet circles today is a six-letter word to describe an activity we have been doing all our lives and which until now had not been elevated to the level that it has been in recent times. S-e-a-r-c-h. We search all the time. It is as basic as breathing. Sometimes, we search our pockets and purses. At other times, we search our memories. A decade ago, we started searching the global web of documents – and then gave up because of the irrelevance of the results. Google changed all that. And in the past year, Search has become the most important word in the online lexicon as Google, having surpassed Amazon and Yahoo, is only slightly behind eBay in the race to become the most valuable company in the Internet space.

Even as Search has become one of the more important activities that we do, let us step back for a few moments and consider the online model of what is happening. We enter a word or two in a keyword box – either on a web page or on our desktop (part of a toolbar or the browser bar). In zero-point-something seconds, the search engine returns to us a set on our computer screen a dozen or so links of matching content with maybe half-a-dozen advertiser links. Think about this: a world wide web of billions of documents distilled down to less than twenty identical links for each of us – irrespective of location and time. This is the world, simplistically speaking, on which multi-billion-dollar valuations have been built.

We all seem to be gleefully clicking away at this narrow set of results – because the advertisers are bidding up what they are willing to pay for our attention. Even as the world becomes richer with personal publishing tools providing a much wider set of amateur publishers, the interface to the web when it comes to search has barely changed. If anything, Search has become the hottest space on the Internet – everyone from Microsoft to Yahoo to Amazon along with tens (or perhaps) of entrepreneurial start-ups are all working to stake out the future. Search has become synonymous with the Internet’s future – and perhaps, computing’ s future.

There is plenty of ideation going around. Will Google do a browser? Will our word processing application be delivered by Google with the right panel – free to cost, and ads filling up the right panel contextually related to what we are writing? Will all the world’s information – books, TV programmes, our own disks – be searchable at the click of a button? Can all this world really be compressed down to a couple dozen links? Is our world really that simple? Are each of us so identical that we can all be delighted with the same set of results? Is Search a momentary interface in time – or is it that all-encompassing window to the world?

I think of today’s Search-mania as good – because it has focused attention on the problem (and we don’t seem to be thinking enough about it). The problem search is trying to address is that of too much data – and that is growing faster than the search engines can index or derive insights into. This mania is also bad because it takes away attention from many other things that are happening in the related, sometimes overlapping worlds of content, mobility and computing. In this series, we will take a look beyond the search core.

Tomorrow: Perspectives

Tech Talk | PermaLink

TECH TALK: The Future of Search: Perspectives

Let us consider some perspectives on today’s search and the world of tomorrow.

John Battelle wrote (September 20, 2004) about a meeting with Raymie Stata of Bloomba, which was acquired by Yahoo:

[Stata] points out that search is not really the big trend of the decade, it's the proliferation of data in the first place. I quite agree, search is our response to the extraordinary info-abundance in which we're all awash. Stata is particularly interested in the "my stuff" problem - integrating search into what we believe is "our" information, and designing interfaces that take that point of view out to the web.

"I see search as falling behind," Stata told me. "So much is accessible now." He continued: "I don't see how traditional search - crawl, take a 2.5 word query, and display ten results - can get much better."

Stata believes search has a user interface problem, to put it rather simply…[There is a ] metaphor that search is the C prompt of the internet, and that the interface is due for an upgrade. "Search is a metaphor," Stata claims, one that users have come to understand, much as they understand nested folders on a computer desktop.

Ramesh Jain wrote in a white paper (August 2004) on next-generation search:

What happened to the web was unexpected and extraordinary. Even Tim Barners-Lee could not have imagined how rapidly the Web grew. But what is happening and what is coming soon is even more mind-boggling than what happened in the past. The volume of data is continuously increasing. More importantly, the amount of live sensory data is increasing very fast. Sensor nets are being designed for applications ranging from understanding birds’ nesting behavior to homeland security applications. Video and audio data is exponentially increasing. With cameras becoming integral part of mobile phones and service providers looking for applications to promote uses of those cameras, completely novel applications are just around the corner. The nature of entertainment is changing fast and that will be integral part of the web. First Personal Video Recorders (PVRs) like TiVO considered themselves a consumer electronic device and did not see any need for internet connectivity. Now cable service providers offer you directly a PVR on line and internet is becoming essential component of PVRs.

It is clear that in comparison to the Web of the last century, the nature of data on the Web later in this decade will be very different in the following aspects:
• Volume of data is growing by orders of magnitudes every year.
• Multimedia and sensor data are becoming more and more common.
• Spatio-temporal attributes of data are important.
• Different data sources provide information to form the holistic picture.
• Users are not concerned with the location of data source, as long as its quality and credibility is assured. They want to know the result of the data assimilation (the big picture of the event).
• Real-time data processing is the only way to extract meaningful information
• Exploration, not querying, is the predominant mode of interaction, which makes context and state critical.
• The user is interested in experience and information, independent of the medium and the source.

Effectively, the nature of the knowledge on the Web is changing very fast. It used to be mostly static text documents; now it will be a combination of live and static multimedia, including text, data and documents with spatio-temporal attributes. Considering these changes, can the search engines developed for static text documents be able to deal with the needs of the Web?

Tomorrow: Perspectives (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Perspectives (Part 2)

Scientific American writes:

New search engines are improving the quality of results by delving deeper into the storehouse of materials available online, by sorting and presenting those results better, and by tracking your long-term interests so that they can refine their handling of new information requests. In the future, search engines will broaden content horizons as well, doing more than simply processing keyword queries typed into a text box. They will be able to automatically take into account your location--letting your wireless PDA, for instance, pinpoint the nearest restaurant when you are traveling. New systems will also find just the right picture faster by matching your sketches to similar shapes. They will even be able to name that half-remembered tune if you hum a few bars.

Future search services will not be restricted to conventional computing platforms. Engineers have already integrated them into some automotive mobile data communications (telematics) systems, and it is likely they will also embed search capabilities into entertainment equipment such as game stations, televisions and high-end stereo systems. Thus, search technologies will play unseen ancillary roles, often via intelligent Web services, in activities such as driving vehicles, listening to music and designing products.

Another big change in Web searching will revolve around new business deals that greatly expand the online coverage of the huge amount of published materials, including text, video and audio, that computer users cannot currently access.

Ironically, next-generation search technologies will become both more and less visible as they perform their increasingly sophisticated jobs…Eventually it will be difficult for computer users to determine where searching starts and understanding begins.

Charles Ferguson in Technology Review: Until now, competition in the search industry has been limited to the Web and has been conducted algorithm by algorithm, feature by feature, and site by site. This competition has resulted in a Google and Yahoo duopoly. If nothing were to change, the growth of Microsoft’s search business would only create a broader oligopoly, similar, perhaps, to those in other media markets. But the search industry will soon serve more than just a Web-based consumer market. It will also include an industrial market for enterprise software products and services, a mass market for personal productivity and communications software, and software and services for a sea of new consumer devices. Search tools will comb through not only Microsoft Office and PDF documents, but also e-mail, instant messages, music, and images; with the spread of voice recognition, Internet telephony, and broadband, it will also be possible to index and search telephone conversations, voice mail, and video files…All these new search products and services will have to work with each other and with many other systems. This, in turn, will require standards.

Tomorrow: Perspectives (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Perspectives (Part 3)

Bambi Francisco writes:

Our own content will likely dwarf what's currently being created by large media and the traditional providers of information.

The abundant flow of digital information on the Web is thanks in large part to all of us who are creating it, from the plethora of blogs, e-mail correspondence, instant messages, to the ever-increasing amount of digital music, photos and videos.

As everything becomes digital, the need to organize the information becomes even more urgent. That's why I believe that organizing our digital world is not a minor feature, as some have said about desktop searching. In fact, these features will be far more useful and addicting than we think.

In a digital world, there is no delineation between video, text, audio and voice. It'll be delivered by anyone. What does it matter who's bringing this information to me or giving me the platform to exchange information on?

Joe Wilcox (Microsoft Monitor): Search is one of several mechanisms (fast data connectivity is another) that could catalyst alternative platforms. Search would give tremendous utility to portable devices connected to the Internet or home or corporate networks. With so much computing focus on information and so much information stored somewhere else (meaning not locally), ubiquitous search could unify the utility of many disparate types of devices. For example, in the advancing communications era, a smartphone could offer Internet search, e-mail, instant messaging and even digital content capabilities like taking pictures without the need for a Windows PC… So like Microsoft integrated the browser into Windows to fight off the threat posed by the Web, so the company is looking to tie the utility of search to its operating system. Because any technology utility where no Windows is required threatens Microsoft's core franchise. And I'm betting some very smart people recognize that search is one of several utilities that could catalyst smaller devices into serious alternative platforms.

BBC News writes:

Search is not just about finding your way around the web. It is now about unlocking information hidden in the gigabytes of documents, images and music on hard drives.

For all these advances, search is still a clumsy tool, often failing to come up with exactly what you had in mind.

In order to do a better job, search engines are trying to get to know you better, doing a better job of remembering, cataloguing and managing all the information you come across.

"Personalisation is going to be a big area for the future," said Yahoo's Yonca Brunini.

"Whoever cracks that and gives you the information you want is going to be the winner. We have to understand you to give you better results that are tailored to you."

This is perhaps the Holy Grail of search, understanding what it is you are looking for and providing it quickly.

John Battelle on his blog about what to expect in 2005: Mobile will finally be plugged into the web in a way that makes sense for the average user and a major mobile innovation - the kind that makes us all say - Jeez that was obvious - will occur. At the core of this innovation will be the concept of search. The outlines of such an innovation: it'll be a way for mobile users to gather the unstructured data they leverage every day while talking on the phone and make it useful to their personal web (including email and RSS, in particular). And it will be a business that looks and feels like a Web 2.0 business - leveraging iterative web development practices, open APIs, and innovation in assembly - that makes the leap.

Tomorrow: Advertising and Innovations

Tech Talk | PermaLink

TECH TALK: The Future of Search: Advertising and Innovations

One of the by-products of the search frenzy is that an interesting new advertising model has developed. Advertising is becoming a service – with the pay-per-click model. Advertisers can decide their budgets and the keywords they want to link their ad to, and they only pay for the ad when someone actually clicks on it. This makes advertising response much more measurable. The New York Times wrote in a story recently:

"In the past, advertising has been hard to track and hard to make accountable," said Tim Armstrong, Google's vice president for advertising sales. Now, he said, advertising has become a dialogue with the consumer.

A TV ad can put the company's name and brand before millions of viewers, said Richard Castellini, the vice president for consumer marketing at CareerBuilder.com. But Google can direct prospective customers to his business by placing a CareerBuilder.com link on the screen of anyone using Google as part of a job hunt.

As for local advertising on radio and television and in newspapers, those media can still be effective ways to blanket large portions of the hometown audience with a uniform message. But individually focused Internet ads are already siphoning business away from locally oriented classified advertising and yellow pages directories - whether in their offline or online forms.

"You're seeing advertising move into advertising that people can seek out, and moving away from mass advertising," said Peter Sealey, a former Coca-Cola marketing executive who now teaches at the Haas School of Business at the University of California, Berkeley. "In the context of that shift, this little niche of Internet search will be a huge beneficiary."

David Jackson: Microsoft and AOL both underestimated the importance and profitability of search. Search benefits from the natural growth of the Internet. One way of looking at this: the search business is the ultimate "user generated content" business. At the same time, the price of pay-per-click (PPC) ads is rising as more businesses establish an online presence, and improved conversion rates boost the return on investment for paid ads. Combine the two, and you find that PPC ad prices are steadily rising, and Google and Overture are sitting on remarkably profitable businesses.

Charlene Li: I believe that in five years’ time that marketers will be buying search and display ads in very similar ways. Marketers will be able to target display ads based on a combination of audience demographics and implied, contextual keywords. They will also be able to buy text ads that appear next to search results based on a combination of keyword search but targeted with different creative for men versus women, or customers versus non-customers.

Next Week: The Future of Search (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Advertising and Innovations (Part 2)

Dave Morgan writes on Sell-Side Advertising: “Advertisers would create large pools of ads designed to deliver measurable performance. They would ‘tag’ those ads with relevant targeting and performance metrics, such as offer, target audience, target context, desired results, timing, desired volume, and price. After selecting appropriate ads and "registering" them with the advertisers, publishers would distribute and target these ads on their sites, leveraging proprietary data about their audience and site to maximize ad performance. Ads that worked well would get more volume; those that didn't would be pulled. In this model, the publisher becomes the marketing service company. It would not only know the true value of its audience to marketers, it would control and own it.”

Christopher Carfi: “It is only a matter of time before the ‘flatness’ of the web becomes mirrored in how people use their local systems, and maybe even in how those systems are organized. With a solid desktop search engine, why should I bother to put things in folders anymore? I can put everything in one place, and the search engine will find it for me. My job just got easier…I no longer think of my machine as a separate entity from the Internet. It just happens to be the nearest node…Of course, this only works well for things that are easily indexable. The images that are fairly flying from camera phones will still need to be indexed, as will the podcasts and the videos and all the other "rich media" out there. That is, until someone figures out a cost-effective way to automatically extract and index metadata from these types or artifacts.”

There is perhaps no other area that is seeing as much excitement and innovation as Search. Rarely a week goes by without some announcement by the majors or the arrival of yet another “Google-killer.” Take a look at some of the recent launches and extensions: MSN Search has launched its new, revamped search engine with its home-grown technology to provide “answers” rather than results, Y!Q from Yahoo offers contextual search and Amazon’s A9 has launched a yellow pages search engine with street photos of millions of businesses. Google also launched a small-business version of its enterprise search appliance. Meanwhile, action on video, local and the desktop search fronts continues. Companies like Gurunet (Answers.com), Blinkx, PubSub, Feedster and Technorati are focused on niches in the search space.

So, everyone seems to be running hard to just stay in place. But are they running the right race?

Tomorrow: What’s Changing

Tech Talk | PermaLink

TECH TALK: The Future of Search: What’s Changing

There have been three versions of search engines in the Internet’s first mass-usage decade. The first “search” was actually Yahoo’s directory – with sites handpicked by editors. This was fine until the number of websites weren’t very large. As the Web grew, the limitations of the directory approach became apparent. Along came Altavista – which used a crawler to get web pages and run indexing algorithms on them. This allowed for keyword-based searching. This era lasted a while until smart webmasters figured out ways to get their pages to show up in the top of the results list by artificially inserting words into their pages.

This problem was addressed to a significant extent when Google launched its search engine using PageRank technology which ranked pages based on incoming links – a measure of authority. This immediately improved the relevance of the results. While there have been some incremental modifications, for the most part, the PageRank technology serves as the base on which most of today’s leading search engines have been built. From Yahoo to Altavista to Google, the focus has been on providing the most relevant results in the quickest possible time to information-hungry users.

In the five years or so since Google’s launch, there have been plenty of new developments in the world and Web around us. As we think of next-generation search, it is important to understand the changing nature of information and usage so we can build up a new model which can then help provide insights into the characteristics of next-generation search engines.

The five most important developments in recent times have been: user-generated content, RSS, mobile phones, broadband and internationalisation. We will look at each of these.

1. User-Generated Content

For much of our history, content has been created by few for consumption by many. This has been because access to the tools for content creation and mechanisms of distribution have been limited. The Internet changed the economics of distribution – anyone could use its global reach to disseminate content. But the tools for content creation were still not easy for mass-market usage.

That has now started to change. Beginning with do-it-yourself publishing via weblogs to image capture via digital cameras and mobile phones, new content is now being created by millions. While the earlier model was that of a “few creating for many”, it is now “many creating for few.” The blog that I create or the photos that you take may be limited to only a very small set of people – but they are people who are important to us.

The latest meme in user-generated content is Podcasting. The New York Times wrote recently: “[P]odcast [is] a kind of recording that, thanks to a technology barely six months old, anyone can make on a computer and then post to a Web site, where it can be downloaded to an iPod or any MP3 player to be played at the listener's leisure…Podcasts are a little like reality television, a little like ‘Wayne's World,’ and are often likened to TiVo, which allows television watchers to download only the programs they want to watch and to skip advertising, for radio or blogs but spoken…And as bloggers have influenced journalism, podcasters have the potential to transform radio.”

Another interesting bottom-up example of user-generated content is the tagging that sites like Del.icio.us and Flickr are supporting. Users can tag any kind of content and then share it with others. Micro Persuasion wrote: "Tags are a natural complement to search because they empower users to create structures that organize unstructured consumer-generated media.”

Tomorrow: What’s Changing (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: What’s Changing (Part 2)

2. RSS

RSS is about syndication and subscription. RSS, which means Rich Site Summary (or Really Simple Syndication), is a format for making content available in a language (XML) which a computer program can read and process. RSS can be used for making available incremental updates available. Interested users can subscribe to be alerted when the updates are available, and can view the updated content in an “RSS Aggregator.”

Mark Pilgrim wrote about RSS in 2002: “RSS is a format for syndicating news and the content of news-like sites, including major news sites like Wired, news-oriented community sites like Slashdot, and personal weblogs. But it's not just for news. Pretty much anything that can be broken down into discrete items can be syndicated via RSS: the "recent changes" page of a wiki, a changelog of CVS checkins, even the revision history of a book. Once information about each item is in RSS format, an RSS-aware program can check the feed for changes and react to the changes in an appropriate way.”

Brad Feld wrote recently: “In the early 1990's, SMTP enabled a raft of companies that built businesses around all aspects of Internet-based email. Shortly thereafter, HTTP enabled - well - an entire industry. SMTP and HTTP are really simple protocols (and - when they were first created - had a slow initial commercial adoptions that suddently went non-linear and became pervasive.) We are seeing exactly the same thing with RSS - and blogging is simply the first broad-based instantiation.”

In the past couple years, there have been a number of RSS-based search engines: Feedster, PubSub and Technorati are some examples.

3. Mobile Phones

The PC is now no longer the only “personal” device in our lives. The mobile phone has usurped the personal space. Mobile phones are moving beyond just voice communications and becoming part of the primary information platform in our lives. 600 million phones were sold globally in 2004 (as compared to about 200 million computers). For many, the phone is the only device that they have – in the emerging markets, the mobile phone is going to become the platform for both communications and computing.

Ramesh Jain wrote recently about home mobile phones is really a “personal computer and communicator.”

The quality and functionality of mobile phones keeps on improving at a rapid rate. Combine this with the increasing availability of video content on phone and increasing bandwidth, and you clearly see what will be the real multimedia client for information and communication technology.

I do believe that for many reasons, mobile phones are real PCs (Personal Computer) or really PCC (Personal Computer and Communicator). Currently computers have evolved to be the most common means for ICT (information and communication technology). This was natural because ICT evolved out of computing. Now we are at a very important point in the evolution of mobile phone technology. Due to miniaturization of sensors, processors, and storage, it is becoming possible to bring enough computing power in mobile phones. This computing power is definitely not comparable to today's computers, but definitely can be compared favorably with lap top computers of a few years ago. The more important thing is that the computing power is becoming enough to utilize mobile phones as a powerful client, more powerful than traditional clients of modern road warriors – the lap top computer. Why do I think that it will is a better device to replace a lap top for many uses?

Are there any inherent limitations in phones? I really don’t think so. The screen size is the one that people will mention. I think that in future screens will be very good quality, of course they will be smaller than your media room's 65 inch TV, but for mobility I am willing to settle for a smaller screen as long as it gives me good quality video. And that is already started – it is not yet there but will be there soon.

As we shall see later, the personal, always-available and always-on nature of mobile phones will change our expectations of search in very interesting ways.

Tomorrow: What’s Changing (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: What’s Changing (Part 3)

4. Broadband

Pipes connecting our computers (and mobile phones) to the network have been growing fatter and fatter. Affordable Broadband is the buzzword even in countries like India. What always-on, broadband does is fundamentally change our expectations of content in three ways: the “network” is always-available and so we turn to it for even the most trivial of queries, the content offering can be beyond text and combine the rich media elements, and it allows consumers to also become producers of content.

In the developed markets, the talk of IP-TV is the talk as phone companies seek to muscle into the territory traditionally dominated by the cable companies. Consider what SBC wants to do in the US (as elaborated in the Wall Street Journal): “SBC wants to fulfill an age-old dream of offering a bundle of consumer services through one high-speed pipe, something that some cable companies are already doing. To catch up, SBC plans to bundle its TV offering with phone, wireless and Internet services in a package that could end up costing about $100 a month. Competing with cable and satellite television, SBC wants to offer viewers the chance to watch TV on demand, rather than at scheduled times, as well as hundreds of channels, many geared to niche audiences.”

As TV becomes digital, it becomes searchable. The likes of Blinkx, Yahoo and Google have launched video search (even though they are quite rudimentary). Another dimension in which broadband will make a difference is the interface that we use to interact with the results of the search – broadband will make it possible to have rich, interactive, video-game-like environments.

5. Internationalisation

The Internet was for long the domain of the English-speaking developed markets. No longer. Even as content in other languages has grown, we are now seeing millions in countries like China and India get online. (The Internet user base in China is estimated to be 80 million, while that in India is put at over 30 million.) English is no longer the first language of the Internet as the non-English-speaking world goes online. This internationalisation of the Internet also brings up a couple of other interesting challenges – and opportunities.

The first deals with the language issue and the availability of content. Ramesh Jain explains: “A serious problem in most of the emerging world that does not speak English and is not exposed to computers as we are is that most, almost all, information is not in the cyberspace and is not likely to get there due to language and resource hurdles. People living in developed countries, particularly countries like USA where I live, assume that every thing important happens in cyberspace. We will have to go beyond that misconception…It will be interesting to think of innovative ways to combine computer network, phone network, and human network to perform searches in real world.”

The second challenge deals with the suitability of the keyboard itself as the right medium of interaction. Once again, Ramesh Jain: “I find it exciting to think that I could browse Internet using voice and get voice or pictorial responses rather than having to read things on a small screen and struggle to type in using a limited keyboard. In fact, a major attraction of advances in phones to me is the possibility of no keyboards. Keyboard is the biggest hurdle in advancement of Internet technology for utilization by people living in all parts of world.”

Tomorrow: Web and Information Models

Tech Talk | PermaLink

TECH TALK: The Future of Search: Web and Information Models

The five trends that we discussed -- user-generated content, RSS, mobile phones, broadband and internationalisation – have profound implications on the way we will access information going ahead. Even though we have come a long towards getting “information on our fingertips,” we are not really there – yet. As we look at the fourth generation of search (after Yahoo’s directories, Altavista’s crawlers, Google’s PageRank), we need to rethink the model of the Web – and the world – around us. This will help us consider what next-generation search will be all about.

Rich Skrenta has kicked off the discussion in this direction with a recent post pointing out the difference between the Reference Web and the Incremental Web:

Google searches the reference Internet. Users come to google with a specific query, and search a vast corpus of largely static information. This is a very valuable and lucrative service to provide: it's the Yellow Pages.
Blogs may look like regular HTML pages, but the key difference is that they're organized chronologically. New posts appear at the top, so with a single browser reload you can say "Just show me what's new."

This seems like a trivial difference, but it drives an entirely different delivery, advertising and value chain. Rather than using HTML, the delivery protocol for web pages, there is a desire for a new, feed-centric protocol: RSS. To search chronologically-ordered content, a relevance-based search that destroys the chronology such as Google is inappropriate. Instead you want Feedster, PubSub or Technorati. Feed content may be better to read in a different sort of client, such as Newsgator, rather than a web browser.

There are 4-8 million active blogs now. At this size, you can still "know" the top bloggers, and find new posts worth reading by clicking around. But when the blogosphere grows 100X or 1000X, the current discovery model will break down. You'll need algorithmic techniques like Topix.net or a Findory to channel the most relevant material from the constant flood of new content.

After I read the post, this was my immediate reaction:

Rich is on the right track, but there are a few additional points which need thinking:

- We need to think beyond just text to multimedia for mass-market content creation and management. [Think Flickr.]
- In emerging markets like India, the mobile and not the computer will be at the heart of the Incremental Web.
- The interface has to go beyond the search box to more natural navigational interfaces. [Think Speech.]
- The published content is being amplified/tagged by the mass market -- this also needs to be taken into account. [Think Del.icio.us.]
- A user's "subscriptions" will be the filter through which the user will want to see the Incremental Web. [Think RSS+OPML.]

Next Week: The Future of Search (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Web and Information Models (Part 2)

Even before we get to constructing a new model for next-generation search, let us look at a few relevant pointers from others.

Andrew Nachison wrote:

[There’s been] a discussion that’s been going of for years among professional journalists and mainstream media executives about the incalculable value of human editors – and the inadequacies of a world experienced only through personalization systems powered by the content judgments of ordinary people rather than professional editors, or through algorithm edits of automated services such as Google News and Topix.

So the issues might be boiled down further:
1. Machines vs. humans?
2. Who profits from the exploding digital datastream?

And maybe there's a third:
3. Who controls the datastream itself?

All information is digital, and from this emerges a limitless potential to parse, format and distribute it; but life is analog. The emerging challenge for society is to seek pathways that bridge growing volumes of data with real life, with humanity. This is why I’m so hung up on the notion of a better-informed society.

Andrew also quoted Steve Gillmor: “RSS has created a new kind of information overload, one where Newton Minnow's vast wasteland of 500 empty channels has been replaced with a million channels of compelling information. RSS is about time, and RSS will win. Attention is about what we do with our time, and attention will win. Friends and family are about who we do it with, and we will all win.”

Greg Linden of Findory added (in a comment on Andrew Nachison’s post): “We think it is too hard for readers to find the news they need. Readers with enough patience or need force themselves to skim tens of sources every day for news that impacts them and their daily lives. Many others resign themselves to remaining ignorant of daily events…Findory aggregates news from thousands of sources and helps readers quickly find the news they need. Unlike other news aggregators, Findory is personalized, learning each reader's interests, creating a different front page for each reader, and helping each person discover news they would otherwise miss…We're convinced that personalized news is a big step toward making news easier to read and keeping people well-informed.”

Richard MacManus added: “The control of content is in one sense moving very definitely towards the consumer, or reader (neither term seems to fit in this age of the read/write web!)…RSS Aggregators and topic/tag feeds are two technologies that in a very real sense give power back to the user. I choose (by subscribing) what content flows into my Aggregator. I choose which of a million niche topics to track by RSS… Google and Yahoo - and apps like Bloglines - are the main tools now for accessing the datastream. Their influence over the datastream is increasingly important.”

Tomorrow: Web and Information Models (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Web and Information Models (Part 3)

An interesting post by Allan Engelhardt from July 2004 provides additional context to the problem:

I do not wish to rely on authors creating their own categories. In my experience, people don't categorise. Getting anybody to document what they are doing is enough of a challenge without bringing up topics like “information architecture”.
That is why I am looking for automatic (unsupervised) text clustering. If I have somebody in London who has great ideas for my retail shops; somebody in Manchester who is experimenting with practical changes to my consumer stores; and a man in Glasgow who would like to promote change in our high-street outlets; how do I enable them to discover each other and work together?

I can not use a standard text classifier on this because I do not have a training set.

An alternative approach explored by people like Matt Mower of eVectors is to assume that 10-20% of people will classify and use that to automatically classify the rest. That is an interesting assumption and a well-understood problem (classify text based on examples) with well-documented solutions from naive Bayes through neural networks and on to support vector machines and similar solutions (the list here in roughly order of increasing performance).
However, the issue is that you are always using “yesterday's taxonomy” to categorise. I am not very interested in this, because chances are that if you have a useful taxonomy then you have existing projects within the organisation dealing with the issues, and promoting existing (funded) projects within a company is a (largely) solved organisational problem.

I'm interested in “tomorrow's taxonomy” to bring together people around new innovative ideas. In the example above, assume that retail stores are a new idea and that the corporate terminology ("retail shops”, “consumer stores”, or “high-street outlets”) has not yet been embedded within the corporate culture. How can I bring together the idea-man in London with the guys in Manchester who can implement them and the manager in Glasgow who can promote the change within the organisation and help it become a change project?

Reprise Media’s SearchViews provides a nice, brief summary of Rich Skrenta’s post: “the Reference Web is goal-directed - it delivers results based on relevancy to a users search. This includes sites and services like Google, Amazon, and IMDB. The Incremental Web is goal-directed as well, but is organized chronologically. This includes subject feed sites like The NY Times, Gawker, and Google News.”

Greg Linden: "Even if you monitor just a few tens of sources, you are facing a daily stream of hundreds or thousands of articles. It's a painful, overwhelming task to manually skim it hunting for relevant content. There is precious little discovery in the current model.”

As I thought more about it (and brainstormed with others), I realised that there was something missing in this picture. Between the Reference Web and the Incremental Web, we need an “Archived Web.” More precisely, the Incremental Web and the Archived Web need to be built around subscriptions, tags and discovery. And outside of these three Webs will be the Community Web – one that is built around our social networks and which no search engine can crawl. Other than the Reference Web, each of the other Webs will be prefixed with “My.” This is what I call the Four-Web model.

Tomorrow: The Four-Web Model

Tech Talk | PermaLink

TECH TALK: The Future of Search: The Four-Web Model

The Reference Web is something we are all familiar with. It is the Web that is available to us via two primary mechanisms: by web address (URL of the site), and via search engines (after crawling and processing). Increasingly, it is more of the latter than the former. Search Engines have become the gateway to the Reference Web – to such an extent that if something is not in there, then we don’t think it exists! Much of this Web is based on documents that have been created and put on the Internet over the years. The size of this Web is expanding continuously – witness Google’s efforts to bring libraries with millions of books online, and Amazon’s online photo-enriched yellow pages.

Next comes the Incremental Web. This Web is the world of “Now.” On the one hand, it comprises the flow of news stories and features as published by the mainstream media. On the other hand, there is the continuum of posts from the “long tail” of bloggers – initially, only text, but now enriched with photos, audio and video. In addition, there is a steady stream of tags by people which serves to provide metadata to existing and fresh content. The Incremental Web is being updated in “real-time” by professionals and amateurs from across the world. Some of this updated content is viewed by millions, while others by a handful from the social network of the person publishing it. RSS subscriptions make it possible to personalise the Incremental Web.

The Archived Web falls in between the Reference and Incremental Webs. In fact, it is an extension of the Incremental Web of a single user – stored in a database for future reference. Another way to view this Archived Web is as the Reference Web seen through the lens of a user’s subscriptions. This Web goes beyond just the desktop – it is not necessarily the content created by a user, but the content that the user has decided to “attend to” via the act of adding a subscription to the RSS feed. Attention may be given “now” through a portal-like interface (My Incremental Web) or “on-demand” through a search-like interface (My Archived Web).

The Community Web is different from the other three Webs in the sense that it does not act on information that exists in cyberspace. Rather, it interfaces with the real-world and builds on a user’s social network. It taps into the “other memory” of friends and family – the memory which is their brain! For the first time, we have devices which can help us tap into people’s memories – via the people themselves. Imagine using our mobile phones as front-ends to tap the information present in our social network – not necessarily as published information, but those tidbits which we continuously gather and file away in some part of our brain. The Community Web could, in theory, have provided the answer to Ramesh Jain’s Agre ka petha query. Thus, the Community Web uses people as sensors into the real world.

The Search game played so far has only focused on the Reference Web. My Incremental, Archived and Community Webs have yet to be tapped effectively. And therein lies the opportunity to build the next-generation search engines.

Tomorrow: Attention

Tech Talk | PermaLink

TECH TALK: The Future of Search: Attention

The Reference Web is common for all of us – and that is where we use today’s search engines. It is only now that tools are available to search a part of “my space” – Google Desktop and Furl are some examples. Search is one of the tools we now use to tackle the challenge of information overload. So, as we look at next-generation search, what we are really trying to think about is how we can manage the continuous datastreams that we are immersed in.

Look at a typical day in our lives. We have a steady flow of emails and IM messages. To reply to some of the emails, we may need to do a search across other emails or documents that we may have written earlier. We also browse the web – going to some sites that are our favourites, and other sites on an as-needed basis, possibly spurred by a thought or a pointer from someone. Increasingly, some of us are setting up subscriptions to sites – via RSS. This provides us the incremental content published by those websites. Taken together, there are many “events” that take place in a day which are either delivered to us over email, IM (and perhaps on the mobile) or are sought by us (news, cricket scores, stock quotes) on a need basis.

Over the past few years, the quantum of information that we are expected to process has gone up by a magnitude – and the time available hasn’t changed. We are expected to be much more efficient in our transactions and yet the tools available have remained nearly static. The email interface is much the same – whether it is an email client like Outlook or Evolution, or web mail via Yahoo or Hotmail. To its credit, Google’s Gmail has tried to innovate on the interface using ideas like labels. The web interface has remained the same – a browser. Most of us now also carry a mobile. We also have access to computers at work and at home. Our data silos have increased. The flow of events has increased. The tools available have not adapted.

The next-generation search engines need to morph into information dashboards – with centralised data stores accessible to us on any device at any time. Think of this as the next version of MyYahoo – understands our context (location and time of day), is built on our preferences (subscriptions), leverages the wisdom of crowds (tags), and allows for serendipity (discovery). There is seamless mobility as I can access this event store and flow from not just any of the computers but also my mobile device.

The commodity that the information dashboards seek to optimise is one that has not increased and will not increase – Attention. Alex Barnett quotes Michael Goldhaber: “..ours is not truly an information economy. By definition, economics is the study of how a society uses its scarce resources. And information is not scarce - especially on the Net, where it is not only abundant, but overflowing. We are drowning in information, yet constantly increasing our generation of it. So a key question arises: Is there something else that flows through cyberspace, something that is scarce and desirable? There is. No one would put anything on the Internet without the hope of obtaining some. It's called attention. And the economy of attention - not information - is the natural economy of cyberspace."

To build the new generation of search engines and information dashboards, we need to combine interface innovation with mobility integration and centre them around Attention.

Tomorrow: Events

Tech Talk | PermaLink

TECH TALK: The Future of Search: Events

The Reference Web is built on the work of others. The other three Webs (Incremental, Archived and Community) is built by us. The Reference Web is a database of all that has happened (and been published). The other Webs are a snapshot of what’s happening – more of a real-time datastream rather than a database, more “here and now” than “then and there.”

The Reference Web is like the ocean of information that is out there. It is free – like oxygen. The other Webs are the oxygen of our personal and social lives. The Reference Web, like oxygen, is a pre-requisite for life – and something we take for granted. It is the body of knowledge that has been created over the centuries. The other Webs are necessary for living – for making life worthwhile. It is being created continuously – by us and our interactions with the world around us. The Reference Web is about information. The other Webs are about Events, Insights and Experience.

Ramesh Jain outlined the ideas of the Event Web in an interview with Gartner:

It started to become more and more clear that because of Gutenberg's revolution, systems designers generally think in terms of alpha-numeric information, but in our lives, there are lots of different data sources - audio, visual, tactile. So if you can explore and find the information you are looking for on your own terms, combining audio, video and other elements, that's experiential computing.

What if you put events on the Web? That means that in place of writing an article about our meeting here and posting it to a page on the Web, we start by placing multiple cameras here, along with some other sensors, and you put the recording on the Web so that people can explore this conversation in the same way as the football game I described. So that at any time they could experience what you and I are doing, see how we are doing it, choose their perspective. So this is the concept that I'm trying to advance. It requires very interesting technological challenges because each search that is done becomes different. Time becomes the most important factor because you are more interested in what's going on right now, or what happened in the near past, etc...In Event Web, the focus is, in place of a document about it, the event itself. Documents provide you information. Event Web gives you the experience.

The idea of events has been championed by Tibco’s Vivek Ranadive for enterprises. Om Malik wrote about Ranadive’s ideas a few years ago: “Vivek ‘real-time’ Ranadive could just be the Peter Drucker of the 21st century. While Mr. Drucker and later Tom Peters’ management principles helped reform the global business landscape, Mr. Ranadive is preaching a new technology mantra - real-time computing - that promises to revolutionize the way global giants conduct commerce…He sees a world which imitates the NASDAQ stock exchange like the traders responding to the events like demand for certain shares, or stock’s movement to news related to a particular company, say for example Juniper Networks. He believes that companies should be responding to the changing chaos of business today in such a manner as well. New information about bloated inventories should result in real-time discounts on products or special offers to customers. Mr. Ranadive describes this as zero-lag, cash-to-cash business, where the time delay between cash spent on producing goods, and the cash received from customers is nearly zero.”

A series of developments in the recent past now has the potential to bring the event-driven, publish-subscribe world to our lives.

Next Week: The Future of Search (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Subscriptions

Subscriptions define our interests. We need to make a conscious decision to subscribe to something – be it a newspaper or magazine, or an emailing list, or even a social group (in the physical world). When we add a “buddy” into our IM list, we are “subscribing” to chat with each other. Because we need to be pro-active about “subscribing” to something, there is an inherent decision that we are making about our likes.

Subscriptions by themselves are not new. Email newsletters and newsgroups have existed since the early days of the Internet. Push, as pioneered by Pointcast, was once seen as a major breakthrough technology. After many false starts, the world of subscriptions is now coming into its own. Among the enabling factors have been the acceptance of RSS as a format for syndication (and subscription), the emergence of user-generated content via blogs which has a narrow field of interest, and the growth of RSS aggregators for viewing these niche content sources.

Kevin Laws wrote about RSS:

Unlike the original push towards push technology, the open RSS standard is already widely adopted. As with the web, this may allow a variety of interesting capabilities which extend RSS capabilities in different directions.

One direction relates to automating tasks for you. This is basically the return of agent technology. Now that a wider variety of web sites are available in machine readable format, it should be possible to tell your computer things like "tell me when an article about gnosticism appears". While this is similar to the stored searches on Google, the fact that RSS aggregators are closer to real-time makes this more valuable. The best analogy is "Tivo for the Web" - specify web sites to definitely "record" and the agent can also record a selection of potentially interesting web posts.

Another direction is enterprise use for RSS. Imagine replacing Microsoft Exchange with an interlocking array of RSS feeds. Each user with Outlook receives their shared calendar, contacts, and other information from subscriptions to RSS feeds. Or they become contributors, sharing one of their calendars with others. I'm sure reading that sentence inspires a host of potential objections for why RSS can not do that. Yet.

Steve Gillmor wrote about how RSS and attention coalesce together:

What if…[we] shift to another architecture–say one that supports lightweight routing of XML fragments around the network in a highly accelerated virtualized kind of digital dial tone infrastructure? Let’s call it RSS, flowing through an attention-based inforouter.

In this alternate universe, user interfaces would be plastic in nature, morphing as data types trigger template switching that routes packets of information through transformation engines based on metadata-driven signals. Charting services are overlaid with ticker text treatments, then piped to handheld devices as a stream, and cached on terminal screens to be called up on demand. Color-coded expert opinions are syndicated to executive information feeds to provide real-time "gut decisions" from consultants and rating data from affinity groups.

…on Planet RSS, the data is transformed through a pipeline of services for delivery to the endpoint renderer. Attention-based preemptive caching reduces latency in bandwidth-constrained or offline scenarios. It’s the virtual rich client, the Really Smart Server.

RSS is the HTML of tomorrow, and Subscriptions will be the Search of tomorrow. RSS is reaching a tipping point – and making its way beyond the early adopters. The potential of RSS goes way beyond just reading blogs – it is a fundamentally different way to consume information. For example, I have subscriptions to over 200 RSS feeds now. When I came across a new source of information that I like, I simply add it to my Aggregator. When that source is updated, my Aggregator notifies me – much like an email folder announces new mail. The challenge now becomes that even tracking 200 feeds is becoming difficult – and so the interface needs to change. Even as Search is the window to the Reference Web, the Aggregator is becoming the window to the Incremental Web.

Tomorrow: Tags

Tech Talk | PermaLink

TECH TALK: The Future of Search: Tags

When I first came across del.icio.us and its use of tags, I found it quite flimsy. But as time has elapsed, I have come to believe that tags have the potential to dramatically change the way we look at information – and in the process, lay the foundation for the Memex. David Weinberger puts in nicely:

Del.icio.us kicked "tagging" into gear by giving us a reason to tag stuff. It's a bookmarking site: If you come across a page on the Web that you want to remember, you post the URL to your personal page at del.icio.us. On the way, you tag it with a word or two that will help you find it among the mass of bookmarks you accumulate on your del.icio.us page. The kicker is that everyone else can see not only what you've bookmarked but all the bookmarks that share a particular tag. You can even subscribe to a tag as an RSS feed. For example, I subscribe to the tag "taxonomy," so every time I go into my blog aggregator, I see a list of new pages to which people have applied that tag. You can also see tagging at work at Flickr, a photo post-and-share site that lets you tag your photos or (with their permission) your friends' photos.

Folksonomies are different in important ways from top-down, hierarchical taxonomies — the shape we've assumed knowledge itself takes.

The old way gets some experts together who create a nested tree of concepts into which everything in a particular domain can be slotted. Think of the Dewey Decimal System. Think of the Tree of Life. The new way invites users of information to add a word or three to the objects they want to find again.

The old way provides the vocabulary we are to use. The new way lets us use our own words.

The old way puts the control of the classification system in that hands of the owners of information classifying it. The new way gives control to the users of information.

The old way creates a tree. The new rakes leaves together.

This is not an either-or. The old way — trees — make sense in controlled environments where ambiguity is dangerous and where thoroughness counts. Trees make less sense in the uncontrolled, connected world that cherishes ambiguity.

In our world of events, we can “publish” tags for things we see and want to keep for future reference. We are doing it for our own good, but if we share it with others, a sort-of emergence effect builds structures that no single individual can. We can also “subscribe” to tags – think of this as a sort-of search across our subscriptions or the wider world outside. RSS ensures that we are alerted any time something new comes across and is appropriately tagged.

Tags are the wisdom of crowds. There is every reason for them not to work. And yet, they do. Along with subscriptions, tags are the other fundamental building block of the event-driven interface of tomorrow.

Tomorrow: Discovery

Tech Talk | PermaLink

TECH TALK: The Future of Search: Discovery

Who doesn’t love the joy that comes with discovery?! We are all so delighted when we find something different from what we started looking for. There is an element of serendipity in there. Technology promises to make this process easier – look at the Amazon book recommendations based on our purchase history.

The same holds true for information. Today, discovery comes either via the search process as we sometimes end up finding things “accidentally” or via recommendations from our social network. Bloggers have made the process of discovery easier – by tracking people we know who have certain interests that overlap with us, we rely on their search processes for recommendation.

One such application that is aiding in the discovery process is Rojo.

Technology Review: Rojo has an RSS feed search function and gives readers the ability to flag stories they find important or interesting. But in enabling users to draw on the insights of friends, family, colleagues, and others in their social networks, Rojo departs from most of the competition. Rojo users can invite others to sign up for Rojo accounts; those accounts are linked, much like the accounts on the popular website Friendster. Rojo users can see what RSS feeds the members of their networks are reading and which stories they are flagging. Network popularity also affects the ranking of results when the user searches RSS feeds. “We all depend on our community for content discovery,” says Chris Alden, Rojo’s cofounder and CEO. “Any successful media service has to tap into that.”

News.com: Like Google's PageRank algorithm and other search engine technologies, Rojo examines the link structure of the so-called blogosphere in order to call attention to blog items and feeds that have proved popular with other readers. Along the same lines, it follows e-commerce sites like Amazon.com in recommending related feeds. And like social networking sites such as Friendster, Rojo narrows down the community of blog readers to those within a user-defined network of friends and associates.

As more and more of our interests and actions are available on centralised servers, the process of discovery will become easier – not just discovery of content, but also discovery of other people with similar interests. In a sense, the information dashboards have to build upon the social networking sites – what our friends and family say means a lot more to us than what someone else says. This creates another layer of search – we can view the world in a series of concentric circles which expand the sphere of search and discovery. We start with what we know and our subscriptions. Next comes the flow of events which match the tags that we have set. Then comes the results of discovery in our social network. And finally comes the big wide world indexed by search engines. This funneling of information is what we use all the time in the real world. The technologies and building blocks now exist to do something similar in cyberspace. The challenge lies in the interfaces that we create to make this process simple.

Tomorrow: Interfaces

Tech Talk | PermaLink

TECH TALK: The Future of Search: Interfaces

Interfaces are critical because that is what users see. Two big innovations of recent times have been around the interface – think Google and iPod. In some ways, the current search interface centred around the keyword box can become a drag for today’s search engines and create opportunity for others who do not have the legacy of worrying about what hundreds of millions of users will think. This legacy-thinking has chained us to the folder-icon interface on the computer desktop for over a decade. Ajax can be the foundation to build next-generation interfaces.

The word has been coined by Jesse James Garrett, who explains what it is all about:

Take a look at Google Suggest. Watch the way the suggested terms update as you type, almost instantly. Now look at Google Maps. Zoom in. Use your cursor to grab the map and scroll around a bit. Again, everything happens almost instantly, with no waiting for pages to reload.

Google Suggest and Google Maps are two examples of a new approach to web applications that we at Adaptive Path have been calling Ajax. The name is shorthand for Asynchronous JavaScript + XML, and it represents a fundamental shift in what’s possible on the Web.

Ajax isn’t a technology. It’s really several technologies, each flourishing in its own right, coming together in powerful new ways. Ajax incorporates:
· standards-based presentation using XHTML and CSS;
· dynamic display and interaction using the Document Object Model;
· data interchange and manipulation using XML and XSLT;
· asynchronous data retrieval using XMLHttpRequest;
· and JavaScript binding everything together.

An Ajax application eliminates the start-stop-start-stop nature of interaction on the Web by introducing an intermediary — an Ajax engine — between the user and the server. It seems like adding a layer to the application would make it less responsive, but the opposite is true.

Instead of loading a webpage, at the start of the session, the browser loads an Ajax engine — written in JavaScript and usually tucked away in a hidden frame. This engine is responsible for both rendering the interface the user sees and communicating with the server on the user’s behalf. The Ajax engine allows the user’s interaction with the application to happen asynchronously — independent of communication with the server. So the user is never staring at a blank browser window and an hourglass icon, waiting around for the server to do something.

We need to think of innovative interfaces – and that is where ideas like Ajax come in. But we also need to think beyond the computer – to the mobile device. This is where speech comes in. Think of an integrated query-presentation interaction environment – and that is where we can learn from video games (and word processors and spreadsheets). As Ramesh Jain puts it in his Gartner interview, the search becomes WYSIWYG – what you see is what you get.

Tomorrow: Information Dashboards

Tech Talk | PermaLink

TECH TALK: The Future of Search: Information Dashboards

As we have discussed, Search is an excellent interface for the Reference Web. What then is the interface for the Incremental Web? Today’s RSS Aggregators are a good start but their limitations become all too apparent as the list of subscriptions grows.

The challenge is to manage this flow of events through the subscriptions. There are projects like AttentionXML which promise to provide a solution to this problem. My belief is that we will need to fundamentally rethink of the interface itself and not just the underlying technology. This is because we consume different information sources differently.

For example, on my aggregator, I may have maybe 20 “A-List” feeds – feeds which I want to see with all the new items outlined – somewhat like Samachar. I may then have another 100-200 “B-List” feeds which I want to see on-demand or via a “river of news” style aggregator. Other feeds may be single-item feeds – the closing stock market index, or the cricket scores, or the weather. In addition, I may have subscribed to a number of “tags” from a multitude of different sources.

In essence, as RSS becomes the de facto standard for syndicating information, how the information is consumed will need to be controlled by the user. This is where the Information Dashboard will come in. It will need to give the user the flexibility to package collections of RSS feeds for viewing for different experiences – these could be based on the device I am using (I may want a smaller subset of feeds on the mobile), or time of the day (I may want a different view in the evening as compared to the morning).

Going ahead, Search will evolve along many dimensions – personalised search, vertical search, local search, and more. But the real reason for search is because there is so much information. Once subscriptions start becoming popular, there will be a need for a new interface for this Web which is built around our lives – combining the Incremental, Archived and Community Webs. This is where the next innovation in computing will happen. This is the future of Search – not directly in the field of search, but in addressing the root of the problem of information overload. This is where Information Dashboards will thrive – built around events, subscriptions, tags and discovery, built with cutting-edge software innovations, available to us on the devices of our choice, and focused around optimizing our Attention. The idea by itself may not be revolutionary. But, like search, the difference will be in how it is executed – and most importantly, the User Interface.

Tech Talk | PermaLink

TECH TALK: The Future of Search: Recent Developments

I had originally intended to complete this series on Search at the end of last week’s column. But a number of recent developments has prompted me to continue the discussion on the future of search. The past week has seen Barry Diller’s InterActive Corp acquire AskJeeves for $1.85 billion, photo-blog Flickr being bought by Yahoo, three media companies took up a majority stake in Topix.net, and HP bought Snapfish. Just a short while ago, AskJeeves had purchased Bloglines. The consolidation in the content and search space has begun.

Ramesh Jain had this to say: “The most lucrative thing is the increasing volume of advertisements on search engines. It is expected that this will keep increasing as more and more people go to search engines for getting information and as search engines become part of TV (or Video) space..Photos are becoming a mainstream business. In cyberspace so far, text was the dominant force. I see that slowly photos, audio, and video are going to challenge the total dominance of text.” John Battelle added that “search and television are going to merge.”

News.com wondered if tagging (as propagated by sites like Flickr and del.icio.us) could upstage search in the future:

Given the billions of files available on the Web, tagging has generally been considered unworkable. Flickr has gotten around the problem by recruiting thousands of people to participate for free. Its loose social framework offers a community that lets people discover, quite serendipitously, interesting photos in the collections of strangers. Without a central body of editors controlling the index, the network also can reveal rare insight into cultural zeitgeists from the people using it.

Finding information in the vast and expanding sea of data online is one of the biggest problems to crack. Creating metadata, or tags, for describing files has long been thought of as a solution for hunting down a range of files on the Web, PCs and intranets, but it has remained an elusive goal. That could explain why tapping Web users' desire to create addictive services or communities is an attractive solution.

Peter Merholz, a founder at Adaptive Path, a user-experience company, said the happy byproduct of Flickr and other "folksonomies" is that there's this global categorization of information. "Flickr is valuable because its creators understood the opportunities on the Web in connecting people. It's the same way eBay is all about leveraging the power of the network," Merholz said.

Future applications for free tagging could include news, blogs, Web and enterprise search. "The future of folksnomies involves meshing these user-generated categorizations with more standardized categorizations, such as the Library of Congress or the Getty Thesaurus of place names, so you could start to connect data to allow more of these associations to be made," Merholz said.

Innovation in and around the search space is happening rapidly. More and more relevant information can now be surfaced – not just via results from search engines, but through the action of bloggers and “taggers.” The discussion around attention, events, subscriptions, tags, discovery and interfaces leading to the information dashboards is not complete. Information Dashboards are just the tip of the information iceberg. What happens when all these information dashboards start interacting with each other? But we are getting ahead of the story here.

Tomorrow: The Messy Web

Tech Talk | PermaLink

TECH TALK: The Future of Search: The Messy Web

Let us start by considering what Adam Bosworth wrote recently, describing a web John Battelle termed as the “Messy Web.” Here is what Adam Bosworth had to say:

I've been complaining about two things on the web for years. Think of the web as the worlds best communication machine. Then the promise should be that anyone can connect to any information or application or anyone else and that any application can connect to anyone or any application or any information. We got anyone to anyone early in the form of email and more recently in the form of IM and of Blogs. IM adds real time communication and presence and Blogs add broadcasting to the world along with a dialog with the world. We got anyone to any application from the esteemed Tim Berners Lee in the form of HTML, HTTP, and URL's which changed our world. I say applications because there wasn't any standard way to ask for information. We got, unfortunately, any application talking to anyone (we call this spam). Web services in one form or another are letting applications access other application although, as I've said elsewhere, I think that the standards are too prolix and that a lot of the action will come out of REST and RSS.

But we didn't get two things. We didn't get a standard way to get information (e.g. a standard query model for sites). And we didn't get people working together in communities to create and construct things with one interesting exception, message boards/groups. Mail was the interface, not the web and not IM.

With [Amazon’s] Open Search the lack of standard ways to get information is, for the first time, beginning to change. There is now a simple but de-facto standard way to start querying sites for information. That's hugely exciting. The current standard is limited, but a great start. And the web is now rapidly becoming the place for people to collaborate. Wiki's are growing like wildfire. Folksonomies(tagging) are causing people to quickly and in an emergent bottoms up way, come together to build taxonomies that work for them and surprisingly rapidly become stable.

To get a glimpse of the future, one need look no further than Yahoo’s soon-to-be-launched Yahoo 360.

Tomorrow: Yahoo 360

Tech Talk | PermaLink

TECH TALK: The Future of Search: Yahoo 360

Yahoo 360 is a way to keep connected to friends with blogs, photos and more. It combines a personal portal with social networking. Charlene Li of Forrester Research had this to say after a preview of the service:

Central to the whole service is the concept that you want to communicate and connect with the people that you already know, rather than try to meet new people. To this end, your home page on the service shows the most recent content published by people within your network. This might be a blog post, a photo album, review, or an updated profile item. This page is constantly refreshed as the people in your network update the information on their spaces. This fundamental concept of linking people through their updated “stuff” is what makes Yahoo! 360 unique – and inherently will drive usage of the service higher than traditional social networks. In essence, the content is being pushed to you by the service.

The profile page contains the usual features from social networking sites friends, profile, lists of things you like to do, where you work/went to school, and groups that you belong to on Yahoo! Groups. But it also excerpts content you’ve created that you want to share with your network. This includes not only a blog, but also photos from Yahoo! photos, reviews created on Yahoo! Local, and LAUNCHcast Stations.

The ability to leverage your network to get something done is what gives Y! 360 the real potential to become something even bigger. At the beta launch, users will have the ability to look narrow local business reviews by their network – a rec’d from someone I know counts for a lot more. Of course, this assumes that people will start creating reviews (a clever way for Y! Local to jumpstart reviews on the service). In the future, I can imagine new modules for job searching, dating, travel planning (“What hotel in Paris would you recommend?”), car buying…the list is extensive. Yahoo!’s (as well as MSN’s and AOL’s) advantage is being a one stop shop in terms of leveraging your network’s knowledge across multiple categories.

The last few years have seen the growth of do-it-yourself publishing on a large scale. Whether it is writing text or sharing photos, podcasts, screencasts and even videos, individuals and groups can now publish and share content easily on the Web. What hasn’t changed significantly, though, is how reading (or viewing) takes place on the Web. While RSS aggregators have made headway, they still remain a niche. MyYahoo has been around for a long time, and now supports RSS. But fundamentally, what is needed is a new way to view the content. This is the direction I was heading in with the Information Dashboard.

Tomorrow: Information Dashboards Rationale

Tech Talk | PermaLink

TECH TALK: The Future of Search: Information Dashboards Rationale

Let us first summarise the discussion leading up to the Information Dashboards, and then we shall consider what lies ahead. I’ll begin by condensing my key points in this series so far.

There have been three versions of search engines in the Internet’s first mass-usage decade. The first “search” was actually Yahoo’s directory – with sites handpicked by editors…Then, along came Altavista – which used a crawler to get web pages and run indexing algorithms on them. This allowed for keyword-based searching… Google improved on the relevance of results PageRank technology which ranked pages based on incoming links – a measure of authority. From Yahoo to Altavista to Google, the focus has been on providing the most relevant results in the quickest possible time to information-hungry users.

In the five years or so since Google’s launch, there have been plenty of new developments in the world and Web around us. The five most important developments in recent times have been: user-generated content, RSS, mobile phones, broadband and internationalisation.

User-generated content: Beginning with do-it-yourself publishing via weblogs to image capture via digital cameras and mobile phones, new content is now being created by millions. While the earlier model was that of a “few creating for many”, it is now “many creating for few.”

RSS: RSS can be used for making available incremental updates available. Interested users can subscribe to be alerted when the updates are available, and can view the updated content in an “RSS Aggregator.”

Mobile Phones: The PC is now no longer the only “personal” device in our lives. The mobile phone has usurped the personal space. Mobile phones are moving beyond just voice communications and becoming part of the primary information platform in our lives.

Broadband: What always-on, broadband does is fundamentally change our expectations of content in three ways: the “network” is always-available and so we turn to it for even the most trivial of queries, the content offering can be beyond text and combine the rich media elements, and it allows consumers to also become producers of content.

Internationalisation: The Internet was for long the domain of the English-speaking developed markets. No longer. Even as content in other languages has grown, we are now seeing millions in countries like China and India get online. English is no longer the first language of the Internet as the non-English-speaking world goes online.

The Four-Web Model: The Search game played so far has only focused on the Reference Web. My Incremental, Archived and Community Webs have yet to be tapped effectively… The Reference Web is built on the work of others. The other three Webs (Incremental, Archived and Community) is built by us. The Reference Web is a database of all that has happened (and been published). The other Webs are a snapshot of what’s happening – more of a real-time datastream rather than a database, more “here and now” than “then and there.” The Reference Web is about information. The other Webs are about Events, Insights and Experience. …And therein lies the opportunity to build the next-generation search engines.

Tomorrow: Information Dashboards Rationale (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: Information Dashboards Rationale (Part 2)

Attention: The next-generation search engines need to morph into information dashboards – with centralised data stores accessible to us on any device at any time. Think of this as the next version of MyYahoo – understands our context (location and time of day), is built on our preferences (subscriptions), leverages the wisdom of crowds (tags), and allows for serendipity (discovery). There is seamless mobility as I can access this event store and flow from not just any of the computers but also my mobile device… The commodity that the information dashboards seek to optimise is one that has not increased and will not increase – Attention… To build the new generation of search engines and information dashboards, we need to combine interface innovation with mobility integration and centre them around Attention.

Subscriptions: RSS is the HTML of tomorrow, and Subscriptions will be the Search of tomorrow. RSS is reaching a tipping point – and making its way beyond the early adopters. The potential of RSS goes way beyond just reading blogs – it is a fundamentally different way to consume information…Even as Search is the window to the Reference Web, the Aggregator is becoming the window to the Incremental Web.

Tags: Tags are the wisdom of crowds. There is every reason for them not to work. And yet, they do. Along with subscriptions, tags are the other fundamental building block of the event-driven interface of tomorrow.

Discovery: As more and more of our interests and actions are available on centralised servers, the process of discovery will become easier – not just discovery of content, but also discovery of other people with similar interests. In a sense, the information dashboards have to build upon the social networking sites – what our friends and family say means a lot more to us than what someone else says. This creates another layer of search – we can view the world in a series of concentric circles which expand the sphere of search and discovery.

Interfaces: We need to think of innovative interfaces – and that is where ideas like Ajax come in. But we also need to think beyond the computer – to the mobile device. This is where speech comes in. Think of an integrated query-presentation interaction environment – and that is where we can learn from video games (and word processors and spreadsheets). As Ramesh Jain puts it in his Gartner interview, the search becomes WYSIWYG – what you see is what you get.

In essence, as RSS becomes the de facto standard for syndicating information, how the information is consumed will need to be controlled by the user. This is where the Information Dashboard will come in. It will need to give the user the flexibility to package collections of RSS feeds for viewing for different experiences – these could be based on the device I am using (I may want a smaller subset of feeds on the mobile), or time of the day (I may want a different view in the evening as compared to the morning).

Once subscriptions start becoming popular, there will be a need for a new interface for this Web which is built around our lives – combining the Incremental, Archived and Community Webs. This is where the next innovation in computing will happen. This is the future of Search – not directly in the field of search, but in addressing the root of the problem of information overload. This is where Information Dashboards will thrive – built around events, subscriptions, tags and discovery, built with cutting-edge software innovations, available to us on the devices of our choice, and focused around optimizing our Attention.

Next Week: The Future of Search (continued)

Tech Talk | PermaLink

TECH TALK: The Future of Search: RSS to OPML

As we have seen in the series, Search will evolve along multiple dimensions. It will become more personal, more local, more vertical. It will also move beyond text to encompass multimedia formats. It will also have better support for mobile devices. Search results will combine matches on our own data stored on local disks (or on the Internet) with the information on the Web. Search APIs will allow developers to build search into applications. Search will thus become part of the tapestry and shift to the background.

What will come to the fore is our continued desire for answers and insights – delivered on time to the device of our choice. “Information at our fingertips” is finally going to happen. One of the key enablers will be Information Dashboards -- built around events, subscriptions, tags and discovery, built with cutting-edge software innovations, available to us on the devices of our choice, and focused around optimising our attention.

To consider how information dashboards will be built, we first need to understand how reading on the Web has changed over the past decade. In the beginning, we had pages and websites built around HTML. As the URLs became too many to remember, we started using bookmarks in the browser. Directories like Yahoo helped us navigate through hierarchies to get us to the sites of interest. Search engines like Altavista and Excite helped us find pages based on keywords. Second-generation search engines like Google improved on the relevance and also simplified the interface. In parallel, portals like Yahoo offered customised “start” pages through MyYahoo and their ilk. Email newsletters delivered updates on sites to our mailbox – and continue to do so. Much of this reading was based on “Pull” – we decided what we wanted to see or search, and then clicked on to it.

In the past couple of years, there have been the portents of change in this model. RSS now delivers updates from a subscriptions list to our aggregators. Even though RSS readers are used by a small fraction of Internet users, Yahoo’s adoption of RSS for MyYahoo and a rapid increase in the websites publishing RSS has helped simplify reading on the Web and is taking it beyond the early adopters. The ease of reading, though, has lagged the progress in publishing. Tools like MovableType and Blogger have made publishing easy. Web-based services like Flickr and del.icio.us have enhanced the publishing and sharing process.

Yet, there are limitations. Even as we talk about Web 2.0, there will be a need to upgrade the content publishing and sharing process. For this, a level of abstraction that is a level above RSS will be needed. This is where OPML (outline processor markup language) comes in. OPML is a mechanism to represent a collection of subscriptions. OPML has an interesting feature called transclusion which allows for “remixing” and repurposing of subscriptions – and therefore, content. Just as RSS allowed us a better way to view content ‘pushed’ from sites, OPML will enable a better way to view a collection of subscriptions. Each cluster can have its own associated view. This is the foundation on which Information Dashboards can be built.

Tomorrow: MyToday

Tech Talk | PermaLink

TECH TALK: The Future of Search: MyToday

Imagine going to your very own information dashboard built out of subscriptions of RSS feeds. Let us call this portal MyToday and compare it with how we would read content in today’s generation of RSS Aggregators, which we will call as MyRSSAgg. (Bloglines is an example of MyRSSAgg.)

In MyRSSAgg, each of the RSS feeds and items has the same view. It is based around RSS. Categorisation is typically done via folders – much like email. As a result, the user has little or no control or the view. It is also hard to share one’s subscriptions with others. This is typically done as a one-time export of the subscriptions as an OPML file, which is then imported. [An overview of OPML can be found in these two older columns – 1 2.]

In MyToday, sets of RSS feeds can be clustered together and viewed independently. This is because the fundamental building block of MyToday is OPML. Allowing multiple views is important because we do not consume each of the RSS feeds in the same way. As I had written earlier: “I may have maybe 20 “A-List” feeds – feeds which I want to see with all the new items outlined – somewhat like Samachar. I may then have another 100-200 “B-List” feeds which I want to see on-demand or via a “river of news” style aggregator. Other feeds may be single-item feeds – the closing stock market index, or the cricket scores, or the weather. In addition, I may have subscribed to a number of ‘tags’ from a multitude of different sources.”

In MyToday, the user can essentially associate views with a set of subscriptions, defined as an OPML file. Each OPML file is a collection of other OPML files (transclusion) and RSS feeds. What transclusion does is to allow dynamic inclusion of RSS feeds. For example, I may have a MyToday page built around India. If you want to construct your own India page, you can “transclude” my MyToday India OPML and add to it your own RSS feeds. This way, whenever I make a change to my OPML, it is automatically reflected in your OPML also. On the other hand, if I had done an export and you had done an import, future changes would not get reflected. Transclusion enables users to identify experts in specific categories and use their OPMLs as building blocks for their own views.

Along with the OPMLs, there will need to be a Page Description Language (PDL), somewhat along the lines of HTML. This will be used to define how the OPMLs are laid out. There can be multiple predefined views which the users can chose from – or they can construct their own. So, A-list blogs may be viewed in their expanded forms, B-list blogs could be viewed in a river flow or as a list of blogs (with a number indicating the number of new items). Some RSS feeds could be single-item feeds, while others could be viewed like bookmarks. This way, users can also define views for the subscriptions they have for viewing on a PC and on mobile devices.

Tomorrow: The Wider View

Tech Talk | PermaLink

TECH TALK: The Future of Search: The Wider View

Information Dashboards are the next upgrade to reading (viewing) the Web. They will be have highly intuitive and interactive interfaces – using Ajax-like remote scripting technologies along with ideas from video games to integrate the query and results display environments. As an idea, dashboards are not new – as an example, we see them in the cars we drive. Many enterprise products also have corporate dashboards or portals which allow users to see a wide variety of information.

What is different about the information dashboards as envisioned by portals like MyToday is that they are almost entirely built around RSS and OPML. As a result, they become the view for the Incremental Web. The Reference Web, on the other hand, is viewed via stored bookmarks, results of searches, and via links that we receive in emails sent by others. The problem in each of these cases is that it is difficult for us to track any changes or new information on the sites we see. As a result, when we like a new site, we can either add it to our own bookmarks or put it on a site like del.icio.us for our own benefit and share it with others. But the granularity of the Reference Web is the URL.

The Incremental Web, built around RSS and OPML, extends this and flips the model from pull to push. Sites make available new information via RSS. We can subscribe to these RSS feeds as and when we come across a new site that we like. The URL of the site no longer needs to be remembered or bookmarked. From then on, the incremental content published by the site flows to our information dashboard – without us ever having to go to that site again to check what’s new. Collections of RSS subscriptions – OPMLs – can be shared with others via transclusion.

In addition, all the RSS items that we receive from our subscriptions can be automatically stored. This becomes our Archived Web. Search should be first done across these items, followed by the Archived Webs of our “neighbourhood.” This allows us to get more relevant results because we’ve already defined what we like in our subscriptions and the flow of RSS items is a reflection of our interests. Storage space is cheap enough now to store everything and anything.

What Information Dashboards ensure is a combination of flow and focus of folk content. Flow means that information gets velocity. The problem we face is not that there isn’t enough information but that the right information is not available to us at the right time. Flow via dashboards ensures that information can be tagged and tapped at the right moment. Filters can also ensure delivery to mobile devices so we have a near real-time view of the world around. Focus is about being able to get top-level views on different topics (on the Web, as well as in the enterprise context) to get a sense of what is happening quickly. Dashboards allow drill-down from top-levels. Folk content is what we are already creating via blogs because publishing tools are so much easier. Folk content can also be created by applications and sensors. Each unit of folk content has a ‘permalink’, with each stream having an RSS descriptor. Once that happens, folk content becomes part of the Incremental Web and ready for viewing and sharing via information dashboards. [My thanks to Ramesh Jain for having coined the phrase ‘folk content.’]

Tomorrow: Information Marketplaces

Tech Talk | PermaLink

TECH TALK: The Future of Search: Information Marketplaces

By changing how information is consumed, Information Dashboards will also transform the way information is published on the Web. Consider corporate websites, for example. Little has changed in the past decade on how websites are constructed. They are monolith content systems, and very hard to change. Websites were built for the era where users would get to them mostly by typing in or clicking on a URL. Later, they were optimised for search engine crawlers. Tomorrow’s websites will need to be built for consumption not via URL-based pull but via RSS-based push.

Tomorrow’s websites will have two parts: a Wiki-style publishing system which allows for ease of publishing, and a set of RSS feeds which track the changes and make the new content available for distribution. More than site design, it is the content that matters – content which needs to be pushed out to interested subscribers in real-time. (Another reason why design is less important is because users view the content in their viewers.)

Once corporate websites start publishing information via RSS feeds and users start consuming it on their dashboards, it will become possible to do matching at the back-end and then alerting users on their dashboards. We are already seeing this happen via next-generation job sites like Indeed.com. This will lead to the creation of information marketplaces.

This is what I had written a couple years earlier:

Imagine if every SME (small- and medium-sized enterprise) can publish an RSS stream (via a weblog) about who they are, their products and services, the new developments at their organisation, their take on industry events, and what they are interested in purchasing. In addition, each of the SMEs should also set up subscriptions – based on what they are looking to buy or sell (by keyword or category) or by a company they would like to track.

So, SMEs do what they would anyways do in search of new business opportunities. By making it easier for them to both publish information and subscribe to relevant information, the Publish-Subscribe Web works as a connector, an information market maker. The product in this marketplace is information; the currency is attention.

In a sense, sites like eBay are information marketplaces. They connect buyers and sellers. In an RSS-enabled world, intermediaries like PubSub.com can provide the matching and notification platforms for events and deliver them automatically to information dashboards. In a world of information marketplaces, reputation will matter – because that is how spam will need to be tackled. Companies and users will be able to build reputation online by also contributing useful information to the marketplace – even as they consume from it. Information marketplaces will help the smaller companies connect with other of their ilk – a difficult problem today because neither has enough money to reach the other.

Tomorrow: Memex

Tech Talk | PermaLink

TECH TALK: The Future of Search: Memex

The combination of RSS, OPML and Information Dashboards will also bring to life an old idea – the Memex, first proposed by Vannevar Bush in 1945. I wrote about “Constructing the Memex” two years ago:

Each of us creates a weblog and a personal directory (MyMemex). In fact, for each of our interest areas, we should create a separate page – with its own blogroll and directory. This is used to help focus the results that we will get on our Mirror Blog.

On a regular basis, we go about updating our weblog. New inputs can come from our own thinking, emails that we get or write, documents that we receive or create, web pages and blog posts that we see and like, subscribed RSS feeds coming in to our mailbox, and perhaps inputs from digital cameras.

As each of us updates our MyMemex or the GroupMemex for the communities that we belong to, the interactions of the Memex with the rest of the ecosystem will result in the constant updation of the MirrorBlog, which will point us to people, ideas and information that could be of use to us. It also captures the state of the world (for example, the day’s headlines and weather) to give a context to the thinking that is happening.

One can think of the Mirror Blog as a “blogdex” for our neighborhood. The current Blogdex (and Daypop) sites show what’s popular across the world of bloggers. More often than not, we are not as much interested as what is being discussed by the rest of the world. Our interest is likely to be greater to see what is being discussed by our friends, and their friends. It is this neighbourhood context that the Mirror Blog will focus on.

We can also go to someone else’s Mirror Blog. For example, if you know that I write on information management, if you are keen to get the wider context on information management, you may decide to come and see the Mirror Blog for my “information management” category on my weblog. It is not something you may do everyday, but every once in a while to keep yourself abreast of the most recent happenings in the space.

Some of the underlying ideas to execute the Memex may have changed, but the building blocks remain the same – blogs, RSS and OPML. What is different is how we assemble these elements together. There is little doubt in my mind that the Web is due for an upgrade – given the spurt in user-generated (“folk”) content and the rise of mobile phones. The Memex is what will emerge as Information Dashboards and Marketplaces become more popular. Thus, the future of Search lies in it enabling the creation of these new platforms to help us tackle a problem which has been with us for a long time – lot of information and limited time. Finally, we have the tools at hand to tackle the challenges. Smarter Search is just a beginning, and the Memex is the endgame. Information Dashboards and Marketplaces make up the middle.

Tech Talk | PermaLink