By Richard Morochove
First published August 19, 1999
Search engines stink. These databases that are supposed to help us find what we want on the Web are failing to do the job.
If you've ever tried to find information using Altavista or one of the other dozen or so big search engines you've no doubt felt the frustration. Your search results swamp you with a tide of irrelevant results and don't give you what you want.
Search engines claim their results are better than ever. They've developed defenses against Web sites that attempt to "spam" the index by loading a Web page with irrelevant terms.
Furthermore, they say many irrelevant results can be eliminated if you adopt advanced search techniques that use Boolean logic to fine-tune your search request. For example, "blue chip" might return information on poker chips. But a search for "blue chip and stocks" would be more likely to return information about investments.
But that solves just part of the problem. There are other reasons search engines don't work.
Search engines don't cover anywhere near the full extent of the World Wide Web. According to a recent study by NEC Research there are some 800 million Web pages available to the public. Popular search engines such as Infoseek, Excite, Lycos, MSN and Yahoo index a minuscule proportion of the Web, less than 10 per cent.
The self-proclaimed largest search engine, FAST Search, wasn't included in the study since it's a recent arrival. Yet even FAST Search claims to have indexed just 200 million pages, only 25 per cent of those available. If you limit your search for Web information to one search engine, you're ignoring most of what's out there.
Size isn't everything. The freshness of the database is also important, since the Web changes so quickly. Yet the NEC Research study blasts the engines for being out-of-date. Adding a new page can take months. Furthermore, the engines index more of the popular, U.S. sites.
According to the authors of the study, "The current state of search engines can be compared to a phone book which is updated irregularly, is biased toward listing more popular information, and has most of the pages ripped out."
At the heart of the problem is that search engines don't make money from searches. The user of the service pays nothing, so the engine attempts to make money by selling advertising on the site. This may take the form of ad banners or text links that could be activated by certain key word searches. For example, if you search for personal computers, you might see a banner advertisement from IBM.
Providing a good search service seems less of a priority for search engines today. There're trying to metamorphose into Web portals that offer you everything you can think of, from free Web-based e-mail to the latest news flashes. Their aim is to increase "stickiness," the length of time a visitor spends on the site.
Offering you a good search service could reduce stickiness. If you find what you seek on the first attempt, you'll be off to visit the new site.
On the other hand, a poor search service means you must try several different search terms or view a number of pages of results. This rewards the poor search service with more banner views and more money from advertisers.
Don't expect to get unbiased search results, either. Excite displays a "Try These First" banner at the head of your results. It leads to sites with advertising or other business relationships with the company.
One search engine, GoTo, earns money by openly selling search results direct to advertisers. Web sites bid for search terms. The highest bidder winds up at the top of the results when you search for the term. GoTo even lists the amount each advertiser pays the service if you click on its link.
This is pretty far removed from the early days of the Internet, when search engines evolved as a free service to help surfers find their way around the Web.
How can you get better search results? Apart from using more specific search terms, you can use a search engine to find a specialized directory in your area of interest. Specialized directories don't attempt to cover the waterfront and can concentrate on a narrow field and do it well. Often a directory will include a review or other comments about the sites it lists.
You can use more than one search engine to expand the database covered by your search. A meta search engine such as Metacrawler makes this easier, since it's set up to automatically query ten search engines at once and return results with duplicate listings removed.
To get a really good search service, I suspect we'll eventually need to pay for it. I predict some enterprising search engine will index most of the Web and let you access its quality database for a cent or two per search. CW
Richard Morochove, FCA, is a Toronto-based computer consultant.
Copyright ©1999 by Morochove & Associates Inc. All rights reserved. This work may not be copied or distributed by any means without our prior written permission.
Air, Hotel and Car Rental deals at HopJet.com
Visit the
ComputerWatch Archive to see more columns
Post any questions or comments about this article to