Web's growth outpaces search engines'

By Peter Svensson The Associated Press

Internet search engines are not keeping pace with the explosive growth of the Web. A study found that search engines – which enable a computer user to find information by typing in a word or combination of words – cover a diminishing fraction of Web pages and take a long time to list new sites.

The most comprehensive engine, Google, covers only about one-sixth of the Internet pages that search engines can reach, the study found. That is down from one-third for the best engine a year and a half ago. Northern Light is closely followed by Lycos and AltaVista.

HotBot, which led with 34 percent coverage in the previous study, was down to 11 percent. The study also found that it takes more than six months on average for a new Web page to make it into a search engine's listings.

The study of 11 search engines was conducted by computer scientists Steve Lawrence and C. Lee Giles at the NEC Research Institute in Princeton, N.J. It was published in today's issue of the journal Nature.

Lawrence and Giles estimated that as of February, the searchable Web consisted of 800 million pages containing more than 6 trillion characters. Their December 2004 survey put the number of pages at about 320 million. By comparison, the 532 miles of shelves in the Library of Congress contain an estimated 20 trillion characters.

Continuous surfing

Search engines use computers called "spiders" that continuously surf the Web. They save each page they visit, then follow the links on the page to find other pages. When a user types in a word, the engine looks in its index to see which pages contain it. A page that's not listed in the index will not be found.

The spiders are more likely to find pages that have more links going to them from other pages. Lawrence said that may make it hard for new sites to make it into search-engine listings.

Lawrence also said search engines may be lagging because their databases become more expensive as they grow, without necessarily creating more advertising profits.

Marc Krellenstein, chief technical officer at Northern Light, said he believes the fraction covered by the search engines is actually larger, because he considers the study's estimate of the size of the Web a bit high.

Krellenstein also questioned whether search engines need to concern themselves with every new site. He said they focus on good sites that are of more interest to users.

"The fact that the Web is growing so fast doesn't mean the quality part is growing as fast," he said.

Search recommendations

Lawrence recommends Web surfers looking for something well-known, like weather forecasts, use engines like Google or DirectHit that rank sites according to their popularity.

"These tend to be good for finding and ranking highly information which is well known and has 2 been around for a while," he said.

If you are looking for something very specific or obscure, he recommends search engines like AltaVista or Hotbot.

Above all, be patient. Do NOT expect to see your website ascend to the top SE rankings overnight. Expect to see your website climb gradually over the first 4-6 months or longer. Do not expect to see rankings before 3 months at all.

©Copyright 2006 QuickByte® Productions
QuickByte is a federally-registered U.S. Trademark, all rights are reserved

Shown above is a segment from our Summary Report showing the top search engine rankings one of our sites earned.

These reports are a valuable tool in monitoring and maintaining the positions we have earned, and are supplied to every client using our Deep Submit service