Evolution of Search Engines

News.com writes about how search has evolved:

Once the primary road signs to navigating the Internet, directories have moved to the shoulder. They are being displaced by algorithmic search tools and commercial services that many people–Abbott among them–now believe do a better job in satisfying Web surfers and advertisers. The transformation is bringing to an end an altruistic era of human editors, who once wielded significant clout in driving traffic to Web sites through recommendations made without regard for commercial considerations.”

The transition has sparked a power shift in the search world that is forcing directory leader Yahoo to reinvent its search business to better compete with an uprising of algorithmic and commercial search providers, most notably Google and Overture Services. In response, Yahoo over the past year has continued to distance itself from its noncommercial directory roots, adding paid search links from Overture, demoting directory listings on its search pages to results provided by Google and scooping up algorithmic search provider Inktomi.

The recent flurry of activity at Yahoo has company watchers wondering what the future holds for the portal’s search tools, and what place, if any, there might be for its once dominant directory.

“In October, Yahoo made the directory secondary to Google,” said Danny Sullivan, editor of the industry newsletter Search Engine Watch. “Suddenly the value of getting listed in Yahoo seemed to disappear. Now, if you’re not listed with Yahoo, it may not matter.”

What Yahoo should do to regain its pre-eminence is to enable individuals (especially bloggers) to create their own directories on the lines of what I wrote earlier today.

The Missing Link In Information Management

When we want to look for information on the web, a few years ago in the early days of the Internet, we used to go to Yahoo and use its directory, navigating through multiple levels to reach the site(s) we wanted. As time passes, we started using search engines – first Altavista and Excite, and now Google. In doing so, we have lost out on something important.

Let us consider Google Search. Of course, it is reasonably accurate in what we are looking for most of the time. Or at least that is what we think because we have no way to tell. But the results are the same irrespective of who does the search. We do not have an easy way of specifying clusters of documents to search, or a time period. In short, what is missing is a “context” for the search.

Navigating through directories like Yahoo also has its problems. There is a single global directory (or at best, country-level directories). Also, they do not take us to the document – they will leave us at the site’s home page. Most of the directories are also not scalable because of their centralisation and manual updation process. In fact, this is what created the opportunity for automatons like Google – the web had simply grown too big.

Into this Search Engine and Directory world have come bloggers. Think of them as a collection of ants, each of which makes its local decisions, and yet as a collective creates structures which no single ant would have been able to “command and control”. In other words, bloggers are creating an emergent system with their individual decisions of what to link to (and what not to link to). Bloggers are putting their own brains, their own knowledge at the centre and creating a nano-version of the Internet around their area of expertise.

There is a problem, though. What we say as a blog is actually a “what’s new” page – this is because it is organised reverse chronologically (by time, the newest entries on top). Yes, many blogs have categories, which is good, but even there, the entries are by date and time of post. What’s missing – even though its there embedded within the blog – is the overall context and perspective that is the blogger’s expertise. What’s missing is an Outline, or in other words, a blogger’s directory of the posts which are there.

Why is this important? When I go to a blog, I am not going there just for finding new links and comments on specific areas. I’d like to get a wider and deeper perspective, because I trust the blogger’s expertise. We like talking to experts because they help in putting things in context, like a good book. There is an introduction, there is a set of key ideas, each of which can be explored further, and there is also an overview of the latest developments. Today, most blogs and bloggers only make visible the last of these – the most recent ideas and news. As a reader, I want more.

As a reader, I want every blog to have an outline, a directory of the posts which provide the context. So, if there is an event or news item, I can now place it in the wider view of things, by just seeing where it is in the directory of items. The blogger has this mental map, it is just not visible on blogs today. The result is that it can make blogs and blogger’s viewpoints hard to understand quickly – one is just seeing a snapshot. It is like reading page of a book at random, without having the benefit of a Table of Contents.

Outlines – or Personal Directories – are the missing link in the information milieu. Imagine if each of us bloggers could create a set of pages which put our writings in context like a directory. So, now, if I wanted to find out more about WiFi or the Digital Divide and if I know that there is an expert in this area, then I can go to that person’s blog, knowing that I will get a complete perspective through the outline and links, rather than just what are the new developments. The blogger already has a mental map – a taxonomy, a context – of the space. With transclusion (the ability to connect and show outlines in place), all these individual outlines could be independently linked together to create paths through the web which a search engine or a directory can never do.

Blogs are thus just the first step. The next progression is for each blogger to put together the directory outlining and linking the posts being done, and connecting to other directories. Dave Winerwrote about this in a post last year.

What’s missing? The language – OPML – is already there. What’s needed is a mass-market outlining tool which can be integrated with blogging. Radio Userland has an outliner. But what’s needed is integration at the blog post level – so that when I am doing a post, besides categorising it, I can also place it appropriately in my directory. Into this ecosystem of personal directories should then come search, and the ability to narrow searches – in a way the RSS search engines are now doing to blogs. They still do not cover verticals or trusted blogs, but that can be expected soon enough.

What Outlines will do is provide a context for viewing information. Instead of just seeing news items as individual specks, we will start seeing the landscape as a whole – through the eyes of the experts. This will create a richer overlay on the world that already exists. The time for Outlines has now come.

India’s Telephone Man

Frederick Noronha writes about Ashok Jhunjhunwala of IIT-Madras, and his efforts in creating affordable telecom solutions in helping bridge the digital divide. His vision of telecom echoes what we want to do in computing:

He believes it makes business sense to provide communication to the poor, but “business has to be done in a different way”.

“In 1987, India opened STD PCOs (inter-state telephone booths) in India. We aggregated the demand of middle and lower middle classes of urban people and provided them shared telephony.

“Today there are 950,000 STD PCOs contributing to approximately 25 percent of the total telecom revenue in the country and serving 300 million people who do not otherwise use telephones. The whole thing makes great business sense,” says he.

Jhunjhunwala wants to replicate a similar phenomenon with the Internet.

“Internet is power. It enables people. It is changing the way we live. Those without Internet will have a tremendous disadvantage as we go on. We would like to see that all villages get reasonable speed Internet connection at the earliest.

“India needs (telecom) products at a cost three-times lower than that prevalent in the West. The simple reason is that affordability in India is much lower,” says Jhunjhunwala.

His vision is to connect 650,000 villages in India with the Internet and use that to double rural gross domestic product. He would also like 200 million telephone and Internet connections in India “at the earliest”.

TECH TALK: Transforming Rural India: More ICT Projects (Part 2)

Setu: An initiative by the Maharashtra government for setting up citizen facilitation centres across the state is Setu. The aim, according to the website is to create foundation for citizen centric e-governance, at district headquarters and subsequently at taluka headquarters. Setu offers single window clearance of 83 important certificates (includes renewal of leases ,permits & licenses), quick redressel of public grievances, common registry of letters, petitions for all sections of the office, On line pendency monitoring of all above.

Akashganga: Based in Guajarat, Akashganga [was] conceived with the objective to spread information technology among rural masses. [It] identified milk cooperative society where each family staying in the village is member & visiting the society to sell milk produced by them (through cattle owned) to the society daily twice. Spread of information & technology initiated with automating their routine milk collection process which otherwise was done manually, beginning with computer and software in regional language, weighing scale, card reader & instrument to integrate the full process. Also developed was an accounting system (Rojmel) to automate and integrate their accounting function in the system. (More)

Educational IT Initiatives

There are several projects centred around Education. These include:

Headstart: It is a project in Madhya Pradesh to provide computer-enabled education and development of basic computer skills for all students in Primary Schools and Middle Schools through the 7000 Jan Shiksha Kendras (cluster resource centres) located in Middle School premises in 48 districts. It will equip every Jan Shiksha Kendra in the state with computer hardware and multimedia software. It repositions the JSK as a media unit capable of providing computer-aided education for the children of the middle school in which the JSK is located and familiarization to computers to all children in primary schools through simple demos and games to excite their imagination. So far, about 2,700 Headstart centres have been set up. (More)

AP Schools: In 2002, the Andhra Pradesh government contracted with NIIT for setting up 663 modern computer classrooms with over 8,000 computers and implementation of computer education in high schools to education over 300,000 students. The five-year project was worth Rs 155 crore (USD 30 million), averaging Rs 90 per student per month. (Details)

Vidya Vahini: The Indian government has launched a project called Vidya Vahini to provide for IT and IT-enabled education in 60,000 schools in India over three years (India has about 1.1 million schools), as part of a Rs 6,000 crore (USD 1.2 billion) project. A pilot covering 150 schools is currently underway. Each of the schools would be provided with a computer lab. The computer lab will also be equipped with Internet, Intranet and television to facilitate video-conferencing, Web-broadcasting and e-learning, according to the Financial Express.

Community Learning Centres: CLCs are being set up the Azim Premji Foundation in rural Karnataka, with the objectives of attracting children to schools, creating excitement in and around the school, Simplifying difficult concepts, thereby making learning exciting and fundamentals strong and building a Sound foundation to IT literacy. A CLC has about 6-8 computers in a Government Higher Primary School, used by children of that school during school hours for learning curriculum through interactive games and exercises. They are Used by the community before or after school hours and during holidays. So far, about 90 CLCs have been set up.

Tomorrow: Sristi and Learnings

Continue reading