Bots & search tools

Bots & search tools<@VM>Comb the Web for sweetheart deals using automated information-gathering sites

These tools can scour the Web
for what you need

By Drew Robb

Special to GCN

Online purchasing, in theory, can deliver the highest-quality products at the best available price because online shopping puts the entire market at your fingertips. The problem is that the Internet contains approximately 575,000 times as many pages as the Los Angeles Business-to-Business Yellow Pages. Your fingertips would have to do a lot of walking to get through them all.

That's where bots come in.

A robot, or bot for short, is a program that acts as an agent for a user or another program, or simulates a human activity.

Robotic software generally is believed to have originated with Eliza, an artificial intelligence program developed by Massachusetts Institute of Technology professor Joseph Weizenbaum in 1966 to study natural language communication between humans and machines. Eliza was programmed to act like a psychoanalyst, answering questions with further questions, and it reportedly could be just as frustrating to talk to as a real shrink.

Bots have come a long way. They provide dozens of different online services, assisting not only with purchasing but also with such things as finding news on specific subjects or keeping track of companies within a specific, regulated industry. Some could be of particular use to government information technology shops and other offices:

Newsbots. Newsbots continuously monitor online news services and notify a user when articles meeting programmed criteria appear. Users specify both the types of news being sought and the sources to be watched to tailor the news to their exact needs.

A programmer can easily set one up to keep the IT department up to date on computer news, one to inform the public relations department of agency press coverage and another to track relevant legislative developments.

Shopbots. Although shopbots are often promoted as last-minute shopping tools, they can be harnessed effectively by government purchasing departments to locate the best price on a Sun Microsystems Ultra 60 Model 2450 server, for example. Some shopbots track the price of an item over time, with graphs showing if the price is going up or down.

Shopbots also can be set to notify you by
e-mail when a product is available at the price you specify, check if the item is available on auction sites, perform automatic price comparisons with other suppliers and compile ratings on the technical support provided by online suppliers.

Because each shopbot specializes in specific types of products, an office might use separate shopbots for computer purchases, travel arrangements and other purchases of services or products. At the very least, they can make it easier to get the requisite three quotes for every purchase order.

Spiders or Web crawlers. Spiders are robots that follow hyperlinks and make copies of the linked pages to create directories that can be used by search engines.

In maintaining a site for an agency or department'whether it's an intranet, extranet or Internet site'a spider can be used to index your own Web site, as well as those of other specified agencies or organizations, to keep the site's search engine completely up to date.

For example, the Occupational Safety and Health Administration works closely with the National Institute of Occupational Safety and Health, and its search engine pulls up documents from both organizations' sites.

Update bots. These monitor various sites and inform the user when information has been updated. Depending on the type of bot selected, the user can be notified by an e-mail message, through a customized Web page or by accessing a file that tracks all changes since it was last accessed.

For example, an update bot can track new information posted by major companies or trade groups of the industries an agency regulates. Similarly, in managing a centralized Web site to which people in different offices can post data, the update bot monitors the Web site for changes to post on the 'What's New' page.

Data management bots. There are several types of data management bots. One stores personal information and automatically inputs it into online forms so they don't have to be filled out manually each time. Besides saving time for one's own staff, the forms could also be offered by an agency's site to improve the public's interface with government.

Other data management bots take documents and mine them to create keyword lists or to generate summaries. These keywords and summaries are then used by search engines or internal document storage and processing systems. This enables users to readily locate data without requiring someone to manually summarize documents.

Intelligent service agents. Intelligent service agents act like human employees to answer frequently asked questions. They are a step up from voice mail information systems in that people can ask questions verbally and get answers to those exact questions.

Although these bots cannot handle all questions, they can reduce the number of calls and e-mail messages agency personnel have to answer.

Bot design bots. Although hundreds of bots are readily available, there also are bots that assist in creating robots or intelligent agents to meet exact needs, which can save programmers time.

There are two excellent online reference sites on bots, explaining how to get them and how they can simplify tasks.

The BotSpot, at, contains articles explaining the different types of bots and how they work, as well as reviews of several hundred of them.

Webcrawler has a database containing technical data about more than 200 Web robots at

With the Internet now approaching 1 billion pages, it is impossible for one person to keep track of what is happening even in one sector of it.

Using robots and intelligent agents, however, makes it possible to get the most out of the vast amounts of data available.

We are still a few years away from Jetsons-like robots that clean the house and do the laundry, but software robots are at least available to help with procurement and other routine tasks.

Drew Robb of Tujunga, Calif., writes about information technology.

SiteType of siteWhat it doesProduct
Active Buyers Guide
Intelligent shopping agentTeaches a user about the features
available in a particular product category and then asks a series of questions to determine which product best fits
General consumer
and business
Auction Octopus
Auction searchSearches auction sites in real timeGeneral consumer
and business
Auction Rover
Auction searchSearches auction sites and provides
links to 850 specialized online auctions
General consumer
and business
Auction Watchers
Auction searchSearches online auction sites for computer
products only
Bidder's Edge
Auction searchSearches 77 auction sites for specified
General consumer
and business
Buyer's Index
ComparisonIndexes items from 16,000 Internet and
conventional retailers; has print catalog
information online
General consumer
and business
CNET Shopper
Computer products
Has product listings and price comparisons
for computer products
Computer products
Consumer World
information site
Provides product information and price
General consumer
and business
buying site
Provides product and company
information and links to suppliers
General consumer,
including banks and
phone service
Frictionless Value Comparison Engine
Intelligent shopping
Has 150 product categories and data on
features of 3 million products; takes user
through a series of questions to determine
the best product for the specified application
and business
InfoSpace Compare Price
Product searchLists companies that offer a specified
General consumer
and business
Product searchHas pull-down menus to specify the type of
product you want; offers option of looking
for product reviews or product prices
General consumer
and business
Does price comparisons, tracks prices over
General consumer,
office and computer
Online Shopping Directory
DirectoryProvides a broad directory of sites with
links to several hundred shopping, software
and reference sites
Consumer, business,
government and
education products
Searches Web sites and print ads for
prices; has pull-down menus for selecting
General consumer,
office and computer
R U Sure Shopping Agent
Price comparison
Searches supported sites for the best price
on an item
General consumer,
office and computer
Conducts product searches among 250
online merchants; produces a directory
of products and prices
General consumer
and business
Provides technical articles and product
comparisons; will recommend products and
explain why, and provides links to suppliers
Computer products
Provides pull-down menus to give product
specs and identifies products that meet
those specs
Computer products


  • Records management: Look beyond the NARA mandates

    Pandemic tests electronic records management

    Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

  • boy learning at home (Travelpixs/

    Tucson’s community wireless bridges the digital divide

    The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

Stay Connected