Topix and Findory

Excerpts from a SearchEngineWatch interview with Rich Skrenta about Topix, which “combines an excellent news search engine with two other hot technologies: local search and personalization. The Topix database includes full text news stories from over 4,000 sources, including a great deal of content that’s difficult to quickly access elsewhere. The real power of this nifty news search engine comes from its easy-to-use pre-built pages that aggregate news and other information into more than 150,000 topic-specific pages.”

Rather than starting with a full web crawl, which has 4 billion+ pages, we started with news, which has 4,000 sources, and is very dynamic and high quality content. We don’t cover everything in the world yet, but we do have every place in the U.S., every sports team, music artist, movie personality, health condition, public company, business vertical, and many other topics.

We developed separate software modules to crawl, cluster and categorize articles. The heart of our system is a proprietary AI categorizer that uses a massive Knowledge Base (KB) to determine the geographic location and subject categorization for each story. The final step is the Robo-Editor, which picks the best stories for display.

We have a commercial feed business for companies that want to enhance their own website offerings with deeply categorized news content. Topix.net offers an extremely rich newsfeed — in addition to the standard URL, title, and summary, we have the latitude/longitude of the news source, the latitude/longitude for the subjects of the story, the prominence of the news source, the subject categorizations, and more. We can also “geo-spin” any subject category, to produce a locally focused version. These features give us a lot of flexibility to customize feeds for clients.

In addition to newspapers, Topix.net is crawling radio and TV station websites, college papers, and some high school papers and weblogs. We’re also crawling government websites with “newsy” public information, such as police department crime alerts, health department reports, OSHA violation announcements, coast guard notices, and news releases from other city, county and state level government entities. We are crawling and including press releases too.

Our focus is on hyperlocal deep coverage of the U.S.. We love police blotters and little papers with extremely local coverage. If your local PTA has online meeting minutes, that’s the kind of source we want to add.

The Seattle Times has an interview with Greg Linden of Findory, “offers free, instant personalization of news searches at http://www.findory.com. It learns from the news you select to read and finds articles that match your interests.”

What’s the point? There’s a glut of news out there, and Findory News is one way to find some focus within that information, Linden said. The Web site keeps a record of the articles you’ve read in the past and uses that information to automatically pick the articles you would likely be interested in.

How does it work? Each visitor is assigned an anonymous identifier, a random number that’s part of a cookie a piece of data that tracks a visitor’s preferences. It associates news searches with the individual identifiers. The service is anonymous in that it doesn’t know anything about users other than the articles they’ve read.

Why news? Lots of companies are developing their own personalization services, and Linden could make a nice living as an executive geek. He said he picked the news business because it has a redeeming social value. “If you make it easier for people to read the news, to spend less time and be more informed, that actually has a lot of value,” he said. “People make better decisions.”

Local Search

Gus Venditto writes:

For consumers, local search will be a time saver, in the house and in the car. But for businesses, it will represent a shift in buying habits that may give the local storefront a chance to regain the ground it ceded to online stores.

The first impact is sure to be Web development for small business. Right now, it’s estimated by the Kelsey Group and ConStat’s Local Commerce Monitor that only 48 percent of small business who advertise have a Web site. Lester Chu, vice president of marketing and strategic planning at Verizon, believes that 60 percent of all businesses don’t have a Web site.

Today, many of those Web-unaware businesses are able to keep their online base covered by buying listings through the Yellow Page directories. For a few extra dollars, 1.4 million businesses who advertise in print Yellow Pages have the option of buying online listings that appear at SuperPages, Yahoo and other portals. And if it weren’t for Google, all local businesses would compete on a level playing field, because they would all have an equal chance to buy their way into the same online directories.

Google’s impact could be seismic because it will rank the pages, and that will re-define the meaning of a good retail location. A small store on a remote side street can build more foot traffic with a good Web site than it could with a busy corner location. All the lower-rent store needs is a better education in the intricacies of search engine rankings.

At the global level, brand marketers will need to pay attention to how the new patterns affect product selection. Online stores allow consumers to select the exact brand and model they want instead of settling for what they found on the shelves. Will local search turn consumers into precision shoppers within the neighborhood? A national brand manager will have to do more than help franchises and regional chain stores buy co-op ads and regional radio. They’ll need to do a better job at helping local dealers show up in online catalogs that are optimized for local searching.

Google as Platform

Jason Kottke adds to Skrenta’s post:

They have this huge map of the Web and are aware of how people move around in the virtual space it represents. They have the perfect place to store this map (one of the world’s largest computers that’s all but incapable of crashing). And they are clever at reading this map. Google knows what people write about, what they search for, what they shop for, they know who wants to advertise and how effective those advertisements are, and they’re about to know how we communicate with friends and loved ones. What can they do with all that? Just about anything that collection of Ph.Ds can dream up.

Tim O’Reilly has talked about various bits from the Web morphing into “the emergent Internet operating system”; the small pieces loosely joining, if you will. Google seems to be heading there already, all by themselves. By building and then joining a bunch of the small pieces by themselves, Google can take full advantage of the economies of scale and avoid the difficulties of interop.

Google isn’t worried about Yahoo! or Microsoft’s search efforts…although the media’s focus on that is probably to their advantage. Their real target is Windows. Who needs Windows when anyone can have free unlimited access to the world’s fastest computer running the smartest operating system? Mobile devices don’t need big, bloated OSes…they’ll be perfect platforms for accessing the GooOS. Using Gnome and Linux as a starting point, Google should design an OS for desktop computers that’s modified to use the GooOS and sell it right alongside Windows ($200) at CompUSA for $10/apiece (available free online of course). Google Office (Goffice?) will be built in, with all your data stored locally, backed up remotely, and available to whomever it needs to be (SubEthaEdit-style collaboration on Word/Excel/PowerPoint-esque documents is only the beginning). Email, shopping, games, music, news, personal publishing, etc.; all the stuff that people use their computers for, it’s all there.

Even though everyone’s down on Google these days, they remain the most interesting company in the world and I’m optimistic about their potential and success (while also apprehensive about the prospect of using Google for absolutely everything someday…I’ll be cursing the Google monopoly in 5 years time). If they stay on target with their plans to leverage their three core assets (which, if Gmail is any indication, they will), I predict Google will be the biggest and most important company in the world in 5-8 years.

Danny Sullivan chimes in with his thoughts on the “Google Desktop”:

Will Google’s new free e-mail system, Gmail, be just the first of many things we’ll see in a new Google Desktop? If so, Microsoft could have a lot more to worry about than just Web search.

Today, plenty of people download mail to desktop-based e-mail programs. But Google might convince some of them to take up its e-mail storage offer.

After all, even if you do have a great way to search through desktop-based e-mail, you might like the idea that all your mail is backed up, stored offsite, and easily searchable from anywhere.

Now, take things a step further. Imagine next year Google provides users with 5, 10, or more gigabytes storage space for personal files.

Got a ton of text documents, spreadsheets, and other material? Push it to us, Google would say. We’ll store it, index it, and make it easy to retrieve what you want. Google already indexes this type of material across the Web and has done so for ages.

As broadband expands, such an idea becomes increasingly more feasible. With it, the notion that Microsoft might trump Google with desktop lock-in becomes less of an issue.

There is an interesting opportunity for Indian broadband companies – think like Google on the platform side, and combine with thin clients (akin to handsets) for users to build an end-to-end alternate computing platform the the mass market (today’s non-users) in India. There are 40 million home users waiting.

The Specific Web

John Battelle points to a Computerworld article on the future direction of search engines:

as powerful as they are, search engines have huge weaknesses. For example, a recent Google search on the word Linux took just 0.4 seconds, but it had 95 million hits. Too bad if the one you need is No. 10,000 on the list.

But researchers are poised to revolutionize search technology over the next few years. The most common thrust is to personalize search engines so that they know, for example, that if you’re an IT professional and you search for mouse, you’re more likely to want information about PC devices than about animals.

Google Platform OS

John Battelle points to Rich Skrenta, and says:”Google is a massively distributed platform with its own OS, and how that has given Google the competitive edge to roll out cool stuff like Gmail. Very Web 2.0 stuff here – the Platform Web.” Writes Skrenta:

Google has taken the last 10 years of systems software research out of university labs, and built their own proprietary, production quality system. What is this platform that Google is building? It’s a distributed computing platform that can manage web-scale datasets on 100,000 node server clusters. It includes a petabyte, distributed, fault tolerant filesystem, distributed RPC code, probably network shared memory and process migration. And a datacenter management system which lets a handful of ops engineers effectively run 100,000 servers.

Google is a company that has built a single very large, custom computer. It’s running their own cluster operating system. They make their big computer even bigger and faster each month, while lowering the cost of CPU cycles. It’s looking more like a general purpose platform than a cluster optimized for a single application.

While competitors are targeting the individual applications Google has deployed, Google is building a massive, general purpose computing platform for web-scale programming.

This computer is running the world’s top search engine, a social networking service, a shopping price comparison engine, a new email service, and a local search/yellow pages engine. What will they do next with the world’s biggest computer and most advanced operating system?

Google = Microsoft of Information

There is little doubt that Google’s ambitions in organising the information space are as gigantic as Microsoft’s in the software space. The recent announcement about Gmail (offering free, ad-driven, web-based email with 1 GB of storage and integrated search built-in) sends a strong message to those who thought that there was an opportunity to be the Google of email. If we assume that software and information are pervasive in our world, the two companies which have to want to continue to dominate these two spaces are Microsoft and Google.

Of course, Google is a long way away from achieving the kind of monopoly has in key software verticals. But the ambition is there to see. Google is following the classic Microsoft strategy – embrace and extend. The fun will start in the coming months as Microsoft launches its own search engine. As it seeks to move from software to information, Google will try to move beyond search into other markets (like email and perhaps, the desktop). This will be an epic contest. The ones who will have all to lose in this – Yahoo and AOL.

Google is leveraging what Bill Gates said a few days – that hardware seeks to be free, along with bandwidth (the free hardware point is something Michael Kanellos disagrees with). For Microsoft, the value lies in software. For Google, the value lies in the services that it provides.

Some other views on this:

Writes WSJ: “Google Inc.’s announcement that it plans to launch a new Web-based e-mail service signals an aggressive step into the home turf of portal giants Yahoo Inc. and Microsoft Corp.’s MSN, threatening to plunder bedrock users from the portals and likely forcing big changes in their e-mail services…Google, no doubt, will change the economics of free e-mail. Mr. Rosing said it has no plans to turn Gmail into a paid service in the future and believes it can support it handily using the same type of targeted text ads now seen on its search-results pages…Search-page advertising companies like Google and Yahoo’s Overture unit are hungry for more Web pages on which to display this increasingly popular form of online advertising. Gmail offers Google a huge new source of Web pages for these ads. The idea is something that Yahoo could conceivably try to duplicate.”

NYTimes: “Google will offer consumers better access to searching their own e-mail and could well upset the industry balance by offering free access to services that previously were available only by paying a monthly subscription fee…The standard industry practice is to offer tiered e-mail services, providing only limited storage free and charging higher fees to users who want to preserve a larger number of e-mail messages for capabilities like online storage. Google, by contrast, is planning a service to be supported by advertising that will permit its users to store very large amounts of mail at no cost…One internal Google study put the operational cost of maintaining e-mail storage at less than $2 a gigabyte, enough to preserve tens of thousands of messages of typical length.”

Danny Sullivan (Search Engine Watch): “Mixing ads with email isn’t new. Free services have long earned by inserting ads into the footers of emails that their users send. But some users might be disturbed by the concept that Google, even if only in an automated fashion, would be essentially reading their mail in order to know what ads to place…Google’s not unique in offering web-based mail search functionality. Yahoo Mail has a similar feature, for example. But the ability to store so much of your mail at Google may give its service a huge advantage if search-based mail reading takes off.”

GlobalSpec and Topix

John Battelle’s Searchblog has a couple of interesting posts on two innovative search-related portals – GlobalSpec’s Engineering Web and Topix. First, Battelle on GlobalSpec:

GlobalSpec is a domain specific (or vertical) search engine. It got its start eight years ago as a classic IT play – take all the catalog-based information about engineering parts – sensors, transducers, etc. – and roll it into a huge, cross-referenced database, which you then distribute over the web. Make money by connecting customers to parts suppliers. Simple.

Over the years GlobalSpec has evolved into a robust community of a million or so engineering types who use it to find and spec parts. That alone is pretty cool (I mean, a million engineers!). But the coolest stuff was just launched: They call it “The Engineering Web” and it’s a domain-specific crawl of the web for engineering information. And not only have they crawled the web (about 100K engineering related sites, so far), they’ve also surfaced invisible web databases not found in mainstream search engines – patent and standards sites, for example, which are walled off by registration and business considerations. Anyone can use the service – it’s not limited to registered users. In essence, GlobalSpec has built a portal that drives traffic and intent through their original database business, in the process building an intelligent island of engineering information that lives in the public sphere. Of course this means they can add AdWord-like functionality, which of course they are working on.

Here’s why GlobalSpec points to some exciting developments in search. Because of its limited domain, GlobalSpec can use relatively simple keyword-based algorithms to surface lists of ideas or terms related to your search. This allows you to refine your search in ways that simply don’t scale in the Googleverse. These related ideas are inferred from the results of your initial query. For example, if you search on “aerodynamics”, you will get “aircraft , Flight Mechanics, Helicopter Aerodynamics, computational fluid dynamics and Theoretical Aerodynamics” as related searches.

The GlobalSpec guys outlined a useful trio of attributes shared by domain-specific search engines like GlobalSpec. First is organization. This is the basic premise of domain specific engines – through organization comes efficient search. Schneiter calls his engine “parametric” – everything in the index is organized against the standards and parameters of the engineering field, making “parametric search” a reality. Second is context. Domains engines are by definition contextual, but GlobalSpec has a drop down menu next to its search box that allows you to contextualize the search even more, across a bunch of subdomains like Products & Manufacturing, Company Name, Application Notes (engineers care about this), Suppliers, and Standards. And third is access. Vertical search products, by their exclusive nature, can provide domain-specific access to the invisible web, in fact, they can enable commercial transactions that otherwise would be impossible (as GlobalSpec does, see here – at the bottom is the option to purchase an engineering standard from a company with a deep database of standards, a database which cannot be accessed via mainstream search engines). To that end, I’d argue that a fourth attribute of vertical search engines is commerce – these engines enable serious, highly efficient closed loop markets.

Next, Battelle on Topix:

Topix is an internet media play. More specifically, its a local advertising media play. The service takes a crawl-and-index approach to a vast array of internet news sources, then runs the resultant stew through a metadata engine which tags every news story with location and subject data. Topix then builds more than 150,000 topic- and location-specific pages, pages that live comfortably between the great gunky mass of search results, on the one hand, and the impersonal morass of most news sites on the other.

At the risk of getting mired in academic debate, one could argue that Topix is a proof point in the semantic web. Topix is not interested in every web result that might be rendered for a search news kentfield, for example. Instead, it searches a limited set of web pages in this case thousands of news sites and then annotates the content of those pages with semantic tags – latitude and longitude, for example. It then machine-generates something of an encyclopedia page for Kentfield, cobbling together news, weather, advertising, police blotters a local newspaper in real time.

This is a sign of things to come. As the Web gets bigger, the backlash against the monoculture of Google will give rise to specialised, vertical search engines. There is an opportunity to create the next-generation information portal on which vertical portals and marketplaces can be built quickly.

China’s Baidu

Xinhuanet writes about China’s answer to Google:

When Google launched its Chinese language advertising service in February, the battle suddenly heated up as domestic search engine operators rushed to get a market share they deserve. Among domestic competitors, Robin Lee’s Baidu seems to be one of the most promising, as the firm began generating profits last year and 80 percent of the revenue came from paid listings.

Robin Li has a position that many people envy as president of Baidu.com, the biggest independent Internet search engine operator in China. The buzz surrounding the company comes as it plans to make an initial public offering (IPO) on the NASDAQ stock market in New York, which some analysts say will result in Baidu being worth as much as US$1 billion.

Set up in 1999 in California’s Silicon Valley, Beijing-based Baidu is China’s most popular search engine, Li said, averaging 30 million text searches a day in Chinese alone – a seventh of Google Inc’s 200 million in myriad languages. Li declined to disclose revenue, but said the number was doubling or even tripling every year. About 80 percent of turnover last year came from sponsored links, where a client pays to have its name and Web link appear at the top of a results list when particular words are searched. Baidu’s quick rise and relatively low rate of cash burn meant it has not needed more venture capital since 2000, when it raised around US$11.2 million in the United States.

However, after leading the search engine market in China for four years, Baidu began to face mounting challenges in 2003.

Domestic firm HC International, which got listed on the Hong Kong Growth Enterprise Market in December, also focuses on search technology as one of its core businesses and formed a China Search Engine Alliance with Sina and more than 100 local websites, as well as over 20 sites from leading Chinese media.

US giant Yahoo acquired 3721 Network Software Co. Ltd, which owns domestic search service provider Beijing 3721, for US$120 million. The world’s most popular search engine provider, Google, was also reported to be looking for domestic distributors.

In the face of these challenges, Li believed Baidu’s focus on a Chinese language search engine service would be its biggest advantage.

“Google provides search engine services in more than 80 languages, but Baidu only focuses on the Chinese language, so its investment on the China market is much less than ours,” he said.

He predicted in late December that China’s search engine market would reach US$50 million in 2003 and US$96-120 million this year.

Baidu gets about 80 per cent of its revenues from paid listings of more than 10,000 small and medium-sized Chinese enterprises, 10 per cent from licensing search software to enterprises, and the rest from licensing technologies to Internet portals in China.

Where is India’s Baidu?

X1 for Hard Disk Search

Walter Mossberg writes:

It’s often easier to search the vast reaches of the Web than to quickly and accurately search your own stored e-mail. This is because the most important e-mail program in use today, Microsoft’s Outlook, really stinks at searching. If you have a sizable amount of stored e-mail, searching within Outlook e-mail is painfully slow, and often inaccurate.

To address this problem, a number of companies have developed add-on programs that search Outlook e-mail. I’ve been testing a new entry in this category, a $99 product for Windows users called X1. This search product doesn’t just do a great job of finding things in Outlook e-mail. It also can rapidly search for any word within e-mail attachments and Outlook contacts.

But that’s not all. X1 also searches for words within e-mail in Outlook Express and the Netscape and Eudora e-mail programs. And it can rapidly search for terms in most types of files you have stored on your hard disk outside of your e-mail. These include word-processor documents, spreadsheets, slide presentations, graphics, database files and more.

X1 handles these file searches much faster and better than the built-in search feature of Windows XP, which I find to be slow and inaccurate, and which can’t search within e-mails.

X1 is a very valuable tool for rapidly unlocking all the precious information on your hard disk, and especially in Outlook e-mail.

Search Engine Flaws

Writes Ramesh Jain:

I feel that these engines are still following a dead-end path. They are pursuing information-centric approach. In information centric approach, the model used is that the user comes to the system with a precise question and the system has answer to that question so it provides the answer. This simple model was good for early databases and even for early search systems. Now, when I type a keyword and the system firehoses me with a list of 5 Million items listed 10 on a page, what am I to do except getting frustrated. Advanced queries are each a new query and as every normal user knows, they are not much help. So what is fundamentally wrong with these search approaches. Here are somethings that I think are fundamental to search in the rapidly expanding cyberspace where current search engines are starting to choke and are ultimately likely to choke and become useless, opening up the space for a new fresh approach by a graduate student, or a start-up, working somewhere maybe even today.

1. The engines have the simple model of information based on keywords. Use of ontological filtering to reduce the list helps but does not really solve the real problem. The real problem is that a set of keywords are not semantically enough to express the context in which search is being done.

2. In most cases people have multidimensional search in mind. The simplest and most common example is seen in LBS (Location Based Services) where a keyword must be combined with location to provide the context. This then gives more meaningful answer to a user. Now LBS is not useful only in the context of mobile phones. The main point is providing dimensionality to search. Dimensionality provides different constraints that further helps in expressing context. Current search engines primarily store a link to the sources containing the keyword. This model is too simplistic to allow and match relations leading to context. By developing more rigorous models and extracting and storing all essential information, search can be more contextual.

3. The search environment based on current approach of specifying keywords and getting a list of pointers in return does not scale-up. People are interested in solutions to their problem, not knowing that 5 Million pages on web contain their keywords. The environment should be more exploration based, rather than query based. In exploration based environment, I will have freedom to slice and dice my results in many different ways to explore and find what I am looking for. Also, this exploration must be multidimensional.

4. Current approach of showing a list is too primitive. We never like to look at a table of stock values over time we look at charts. Two-dimensional data (stock value, and time) when reduced to a liner representation, theoretically contains the same information but practical is useless as the number of data item increases. After about 100 items, it is for all practical purposes completely useless. We need to develop powerful visualization mechanisms. This is not a simple point. Mathematically we know that when a two-dimensional information is projected onto one-dimensional space (extend it to higher dimension) there is loss of information. Same is true for human mind in fact that is more true for human mind.