Category: Search Engines

Andy Beal writes about search engines and interviews Microsoft’s Robert Scoble.

An idea from Andy: “Google has the technology to really take advantage of search within email. Why else would it even consider entering this arena? Imagine that, in order to use a free Google email account, you allow Google to provide advertisements and track your email activities. Google could change the way that search results and ads are displayed to free email users. For example, let’s say you receive an email from your brother, the content of which, among other things, gloats about the brand new P4 desktop computer that they just purchased from Dell. As part of the interface you use to read that email, Google magically displays paid search advertising for desktop computers, including a link that will take you directly to the appropriate page on Dell.com. This information would be quite beneficial to you, as you may be interested in seeing how you too can be a proud owner of a P4 computer. Fantastic targeted advertising for Dell, as they know that if you click on the listing, they are halfway there to converting you into another satisfied customer…This idea is so much closer to reality than you may think. Google already has the advertisers with its AdWords service boasting 150,000 users, eager to spend their advertising dollars. It also has the technology to determine which results to show you within your email interface. Google’s AdSense can provide the contextual ad technology that would scan an email’s content to determine which ads are the most relevant to display.”

Scoble: “For Internet searches, I see social behavior analysis tools like Technorati becoming far more important. Why? Because people want different ways to see potentially relevant results. Google took us a long way toward that future as their Google’s results are strongly influenced by how many inbound links a site has. But, now, let’s go further, even further than Technorati has gone. Let’s identify who really is keeping the market up to date on a certain field and give him/her more weight…I also see that search engines that search just specific types of content (like Feedster) are going to be more important (Feedster only searches RSS and Atom syndication feeds)…Oh, and users are going to demand new ways of exporting searches. Google showed us that with News Alerts. Enter in a search term, like ‘Microsoft’ and get emailed anytime a news source mentions Microsoft. Feedster goes further than that. There you can build an RSS feed from a search term. I have several of those coming into my RSS News Aggregator and find they are invaluable for watching what the weblogs are saying about your product, company, or market. For instance, one of my terms I built a feed for is “WinFS” — I’ll be watching to see how many people link to this article and if any of you have something interesting to say I’ll even link back.”

Jeremy Zawodny has an interesting perspective on why Google needs Orkut, the social networking site that was recently launched:

What problems might Orkut solve that Google would otherwise find significantly more challenging?

Those that do seem to speculate about “applying your social network to search” and other exotic stuff, but I’m thinking of something far more basic than that: users.

That’s right, users.

Now, I know what you’re thinking. “Google has millions of users!” Of course they do. How much to they really know about those users? Do they really have a user database from which they can mine interesting data? If they have one, it must pale in comparison to what Yahoo, AOL, and MSN have.

Rather than try to convince users to start “registering” for Google, why not piggyback on one of the most viral fads going around: a social network application? And, for added effect, make it an invite only system so that you feel special once you’re invited.

Just think about it for a few minutes. If you’ve been thru the Orkut registration process, you know that it attempts to collect a ton of data about you. The kind of demographic data that marketing folks drool over. And right now there are lots of folks dying to get that special invite and begin the sign-up process.

Still with me? Good.

Let’s assume that Google internationalizes Orkut and lets it run to the point that it has millions of users registered and active. That’s not an unreasonable thing to expect. Then, one day down the road, they quietly decide to “better integrate” Orkut with Google and start redirecting all Orkut requests to orkut.google.com.

Bingo!

Suddenly they’re able to set a *.google.com cookie that contains a bit of identifying data (such as your Orkut id) and that would greatly enhance their ability to mine useful and profitable data from the combination of your profile and daily searches.

Of course, we know that the “big three” already do this sort of thing to some degree or another. But Google’s lack of intimate knowledge of their users is surely holding them back from doing some of the things they’d love to do. Many folks think it’s just a matter of time before they try to get “real” users signed up.

Adds Dave Winer: “[Orkut is Google’s] identity system. At some point they’ll add a web services interface so our comment systems can connect to their back-end to validate users. Now you can go to one place to see all your comments. Then it gets better. Give it your credit card info, and then when you go to an Orkut-enabled e-commerce site, you can have one-click ordering (modulo a certain patent). Think about all the relationships Google has with sites that run their ads. Even I run their ads on one of my sites, and it’s a pretty good deal, that one site pays for the bandwidth on all my sites. Anyway, that’s a ramble. The net-net — it’s Google’s identity system, and if you trust them, it can be yours too.”

Its a compelling explanation, considering that the registration process and user profiles are what’s missing from Google (as compared to Yahoo or MSN).

The drums have started sounding. The war is coming. NYTimes writes on what to expect in a battle that is likely to rival the Microsoft-Netscape one:

Today, nearly everyone in Silicon Valley, from venture capitalists and chip engineers to real estate agents and restaurateurs, has begun to ask: Will Google become the next Netscape?

[Google’s success] has ignited a three-way battle among Microsoft and its two Silicon Valley rivals: Yahoo, based in Sunnyvale, Calif., and Google, whose headquarters are nearby, in Mountain View. Underscoring the importance of search engines to Internet advertising, Yahoo recently said it planned to end its exclusive reliance on Google for search results and had established its own research lab to try to cut its new rival’s lead.

Google’s financial success is clear. In 2001, the company had virtually no revenue; in the past year, it recorded sales of almost $1 billion and profits of about $350 million, according to several executives familiar with the company’s private financial figures.

Later this year, Microsoft is expected to unveil its own search technology, which Mr. Gates says will help Microsoft catch up with Google. Last week, Microsoft released a test version of a special set of software buttons for its browser designed to direct users to its MSN search and related services. For Google, though, the greater threat is that Microsoft will decide that Internet search, like the Web browser before it, should be an integral part of future versions of the Windows operating system.

For all of Google’s hyperactivity, there is still a lingering sense among many Silicon Valley veterans that they have seen this movie before. The company may not have Netscape’s arrogance, but it is still not clear that all of its clever marketing, technology and brand identification can withstand Microsoft’s onslaught when it arrives.

Google has had plenty of time to learn the lessons. My feeling: it should not do an IPO for the next year. Instead, it should focus on its new technologies and the momentum that is has built. The IPO will shift focus, and that should not happen at such a critical juncture. Google is changing enough cash through its operations – it doesn’t really need the money.

Rafe Needleman writes:

If you want to find a specialized electronic component for your engineering project, you can search the Web for it, but that can be a frustrating experience. While almost all component manufacturers put their catalogs online, they don’t do it in a consistent, data-driven way that helps people looking for parts, much less looking to compare them between manufacturers. While the data is all on the Web, it often doesn’t seem like it, since it is so hard to find.

So GlobalSpec has created a system that collects information about parts from 10,000 catalogs that list about 50 million different parts and services, from transistors to air compressors to contract machine shop services. GlobalSpec president John Schneider thinks that of all the parts an engineer could order for a project, he has about 20% accounted for in his system.

Compared to Froogle or a site like NexTag, GlobalSpec’s difference is the granularity of its data. Compared to Ariba, GlobalSpec is less ambitiousit’s just a catalog aggregator. But that may be just what engineers need. If you’re looking for a control valve, you can specify the range of maximum flow it has to handle (in your choice of 15 units of measurement), as well as precisely what centering mechanism (spring centered, spring offset, etc.) you’re looking for.

It is, in other words, a fairly typical online publishing enterprise, combined with some well-understood search engine economics. But I find it interesting because lately I’ve been seeing more and more specialized search tools and catalogs.

Despite the many advances in Web searching, and in the protocols and standards that make it easier to publish information on the Internet, there seems to be a growing need for specialized directories to make sense of all this information.

Specialty (vertical) search engines look like the next niche.

CNN suggests that Google may be planning an email service and morphing itself into a portal:

Adding an e-mail service would provide a potential boost to Google as its technology lead in the search market seems destined to narrow and it prepares to answer to growth-hungry shareholders, analysts said.

“I’m sure Google is getting more and more concerned about locking in users. It wouldn’t surprise me if they did something very sophisticated with e-mail,” said Danny Sullivan, editor of SearchEngineWatch.com, who tracks the industry.

By moving into e-mail — the Web’s most-used program — Google would open up a huge new market for its lucrative “sponsored links” advertising business that delivers ads tied to keywords in Web searches or on content pages, analysts said.

“If they were to go the e-mail route they’d have to provide an offering that competes with free [e-mail]. Anti-spam is one form of strong differentiation,” said Jim Pitkow, chief executive of Moreover Technologies.

Google’s ambitions are obviously increasing. It needs to cover as many bases as possible before its IPO. The temptation is now to do many things – what else would one expect from a company which is likely to have USD 2-4 billion in the bank shortly? I feel that these are distractions – there’s plenty to be done on Search itself. I will be writing a Tech Talk series on Seacrh from the Indian market perspective next week.

Three articles on the action in the search engine space:

News.com writes about how “some major Web portals (Microsoft, Yahoo and EarthLink) are trying to get an edge in the competitive Internet search market by thinking outside of the browser…and providing access to online searches through a ‘taskbar’ displayed to the side or at the bottom of a PC screen…Advocates of the technology say it offers a simple and direct way for people to call up information such as stock quotes, dictionary definitions and other data without going through the added step of opening a full browser window.”

A second report in News.com discusses Yahoo’s plans to allow external RSS feeds to be added on to the MyYahoo page: “RSS would let MyYahoo users transport feeds from third-party content sites onto their personal pages, intermingling outside links with tailored news, video and financial information from Yahoo. The outside links then direct the reader to content on third-party pages. Such a layout would be a first for Yahoo.”

WSJ reports on Google’s continued expansion of what the search box can do: “Google expanded the types of information that Internet users can search for on its Web site to include such things as area codes, product codes, flight information, vehicle identification numbers and U.S. Postal Service tracking numbers…Computer users, for example, can type in an area code in the search query bar and the top result will show a map of that geographic area. Users can also plug in a vehicle identification number into the search query box to get a link for a Web page with more information about the year, make and model of a specific type of car…Google sees its mission as connecting Internet users to the world’s information, which it hopes to organize and make more accessible.”

So, what it looks like is that we can expect a Unix-style command line interface to search engines, accessible via a desktop taskbar.

Wired News wrote recently about a set of visualisation tools which can help refine search results, starting with the premise that “as wonderful as Internet search engines are, they have a pretty big flaw. They often deliver too much information, and a lot of it isn’t quite what we’re looking for. Who really bothers to read the dozens of pages of results that Google generates?” which creates an opportunity for “software now emerging analyzes search results and automatically sorts them into categories that, at a glance, present far more information than the typical textual list.”

Vivisimo sells its technology to companies and intelligence agencies, and offers free at Web searches. Co-founder Valdes-Perez describes his company this way: If the Internet is a giant bookstore in which all the books are piled randomly on the floor, then Vivisimo is like a superfast librarian who can instantly arrange the titles on shelves in a way that makes sense.

Consider it a 21st century Dewey Decimal System designed to fight information overload. But unlike libraries, Vivisimo doesn’t use predefined categories. Its software determines them on the fly, depending on the search results. The filing is done through a combination of linguistic and statistical analysis, a method that even works with other languages.

A similar process powers Grokker, a downloadable program that not only sorts search results into categories but also “maps” the results in a holistic way, showing each category as a colorful circle. Within each circle, subcategories appear as more circles that can be clicked on and zoomed in on.

Groxis, the 15-person company that introduced Grokker last year and released an upgraded, $50 second version in December, is not out to replace Google. Grokker is not in itself a search engine — it only analyzes and illustrates search engines’ results.

Another visualization possibility is offered by TouchGraph, which has a Google plug-in that shows links as an interconnected web, an appropriate image for the World Wide Web.

Search needs some innovations – I’ll be writing about it in a future Tech Talk series.

SearchEngineWatch writes about the potential in the SME market:

The answer, as one might expect, is revenue-or more precisely, potential revenue. When you add up the number of all the paid search advertisers in the world right now, the total is approximately 380,000. Yet, there is substantial overlap among the advertisers of the different paid search networks. (It’s rare a company that uses Google but not Overture and vice versa.) So, as a rough estimate, the figure is probably closer to 250,000 paid search advertisers on a global basis.

By contrast, the U.S. alone has about 10 million small- and medium-sized enterprises (SMEs), and there may be as many as 30 million more such businesses in developed countries around the world. In the U.S., most of those SMEs, the bulk of which have fewer than nine employees, conduct the majority of their business within 50 miles of their locations.

The model to look forward to is Citysearch which marries search and directory. “Citysearch presents a hybrid approach that combines pay-per-click advertising for local businesses, with a keyword search-driven interface over a structured local database. That provides the ease of use of search with the reliable structured data offered by Yellow Pages.”

WSJ reports on Yahoo’s plans to drop Google as its primary search technology as part of a larger plan to take on Google:

First, Yahoo is expected to dump Google as the primary search technology on its site within a few months, a move that could come around the time Google is preparing a long-awaited initial stock offering. Some marketing firms, which help advertisers manage their online campaigns for search-related ads, say they have been told Yahoo will switch from Google to its own technology as early as the first quarter.

Second, Yahoo wants to combine personalization and customization features to extend the usefulness of searches. Third, it plans to expand its use of “paid inclusion,” whereby it frequently surveys a participating merchant’s sites for the most up-to-date information and includes those findings in users’ search results. Merchants pay Yahoo from 15 cents to more than $1 when visitors click on a link for that merchant.

The strategy is not simply to match what Google does now but to add features its rival can’t easily match. “We’re not going to beat the competition by being the competition,” says Jeff Weiner, the company’s senior vice president in charge of its search and marketplace services. “We’re going to beat the competition by being Yahoo.”

Yahoo isn’t discussing many of its search plans in detail. But some steps toward independence can already be seen on the shopping section of Yahoo’s site, which is now using Inktomi’s technology. Type in “digital camera,” for example, and the site shows pictures of specific cameras along with their prices, flanked by “shopping tools” that allow users to quickly call up price comparisons, fuller specifications and user reviews. By contrast, the same search on Yahoo’s front page, which still uses Google’s technology, returns a familiar text-based list of links, starting with those sponsored by retailers and followed by other camera-related sites ranked by popularity.

In the future, Yahoo officials say, searches could become much more personalized. They could be tailored to return results that reflect users’ past Web-surfing behavior, for example, or preferences or interests they list in a profile.

News.com has more on Yahoo’s search plans. Yahoo, Google and Microsoft are the three players to watch in the search engine space. The period leading up to Google’s IPO is likely to see a lot of action.

Tim Bray (who is looking for something new to do) has a wishlist for what the next-generation search software should be as part of his Search Series. He calls it BRF and elaborates:
– is Open-Source
– is Web-Centric
– Interfaces to Everything
– is Part of Apache
– is Internationalized
– Comes with a Bunch of Document Readers
– Comes with a Robot
– Comes with a Filesystem Walker
– is Self-Managing
– Keeps Running
– is XML-Capable
– is Fast
– is Ready for PHP, JSP, and Friends
– Has Programmmable Ranking
– Does Booleans, Phrases, And So On

Tim would like to spend the next year building the BRF out…sounds like something good to be a part of. His ideas could be combined with Steve Gillmor’s Information Router built around RSS, and we could have the base for a new Information Platform.