Bots & search tools

Bots & search tools<@VM>Comb the Web for sweetheart deals using automated information-gathering sites

• May 24, 2000

These tools can scour the Web
for what you need

By Drew Robb

Special to GCN

Online purchasing, in theory, can deliver the highest-quality products at the best available price because online shopping puts the entire market at your fingertips. The problem is that the Internet contains approximately 575,000 times as many pages as the Los Angeles Business-to-Business Yellow Pages. Your fingertips would have to do a lot of walking to get through them all.

That's where bots come in.

A robot, or bot for short, is a program that acts as an agent for a user or another program, or simulates a human activity.

Robotic software generally is believed to have originated with Eliza, an artificial intelligence program developed by Massachusetts Institute of Technology professor Joseph Weizenbaum in 1966 to study natural language communication between humans and machines. Eliza was programmed to act like a psychoanalyst, answering questions with further questions, and it reportedly could be just as frustrating to talk to as a real shrink.

Bots have come a long way. They provide dozens of different online services, assisting not only with purchasing but also with such things as finding news on specific subjects or keeping track of companies within a specific, regulated industry. Some could be of particular use to government information technology shops and other offices:

Newsbots. Newsbots continuously monitor online news services and notify a user when articles meeting programmed criteria appear. Users specify both the types of news being sought and the sources to be watched to tailor the news to their exact needs.

A programmer can easily set one up to keep the IT department up to date on computer news, one to inform the public relations department of agency press coverage and another to track relevant legislative developments.

Shopbots. Although shopbots are often promoted as last-minute shopping tools, they can be harnessed effectively by government purchasing departments to locate the best price on a Sun Microsystems Ultra 60 Model 2450 server, for example. Some shopbots track the price of an item over time, with graphs showing if the price is going up or down.

Shopbots also can be set to notify you by
e-mail when a product is available at the price you specify, check if the item is available on auction sites, perform automatic price comparisons with other suppliers and compile ratings on the technical support provided by online suppliers.

Because each shopbot specializes in specific types of products, an office might use separate shopbots for computer purchases, travel arrangements and other purchases of services or products. At the very least, they can make it easier to get the requisite three quotes for every purchase order.

Spiders or Web crawlers. Spiders are robots that follow hyperlinks and make copies of the linked pages to create directories that can be used by search engines.

In maintaining a site for an agency or department'whether it's an intranet, extranet or Internet site'a spider can be used to index your own Web site, as well as those of other specified agencies or organizations, to keep the site's search engine completely up to date.

For example, the Occupational Safety and Health Administration works closely with the National Institute of Occupational Safety and Health, and its search engine pulls up documents from both organizations' sites.

Update bots. These monitor various sites and inform the user when information has been updated. Depending on the type of bot selected, the user can be notified by an e-mail message, through a customized Web page or by accessing a file that tracks all changes since it was last accessed.

For example, an update bot can track new information posted by major companies or trade groups of the industries an agency regulates. Similarly, in managing a centralized Web site to which people in different offices can post data, the update bot monitors the Web site for changes to post on the 'What's New' page.

Data management bots. There are several types of data management bots. One stores personal information and automatically inputs it into online forms so they don't have to be filled out manually each time. Besides saving time for one's own staff, the forms could also be offered by an agency's site to improve the public's interface with government.

Other data management bots take documents and mine them to create keyword lists or to generate summaries. These keywords and summaries are then used by search engines or internal document storage and processing systems. This enables users to readily locate data without requiring someone to manually summarize documents.

Intelligent service agents. Intelligent service agents act like human employees to answer frequently asked questions. They are a step up from voice mail information systems in that people can ask questions verbally and get answers to those exact questions.

Although these bots cannot handle all questions, they can reduce the number of calls and e-mail messages agency personnel have to answer.

Bot design bots. Although hundreds of bots are readily available, there also are bots that assist in creating robots or intelligent agents to meet exact needs, which can save programmers time.

There are two excellent online reference sites on bots, explaining how to get them and how they can simplify tasks.

The BotSpot, at www.botspot.com, contains articles explaining the different types of bots and how they work, as well as reviews of several hundred of them.

Webcrawler has a database containing technical data about more than 200 Web robots at info.webcrawler.com/mak/projects/robots/active.html.

With the Internet now approaching 1 billion pages, it is impossible for one person to keep track of what is happening even in one sector of it.

Using robots and intelligent agents, however, makes it possible to get the most out of the vast amounts of data available.

We are still a few years away from Jetsons-like robots that clean the house and do the laundry, but software robots are at least available to help with procurement and other routine tasks.

Drew Robb of Tujunga, Calif., writes about information technology.

Site	Type of site	What it does	Product specialty
Active Buyers Guide www.activebuyersguide.com	Intelligent shopping agent	Teaches a user about the features available in a particular product category and then asks a series of questions to determine which product best fits	General consumer and business
Auction Octopus www.auctionoctopus.com	Auction search	Searches auction sites in real time	General consumer and business
Auction Rover www.auctionrover.com	Auction search	Searches auction sites and provides links to 850 specialized online auctions	General consumer and business
Auction Watchers www.auctionwatchers.com	Auction search	Searches online auction sites for computer items	Computer products only
Bidder's Edge www.biddersedge.com	Auction search	Searches 77 auction sites for specified items	General consumer and business
Buyer's Index www.buyersindex.com	Comparison	Indexes items from 16,000 Internet and conventional retailers; has print catalog information online	General consumer and business
CNET Shopper www.shopper.cnet.com	Computer products shopper	Has product listings and price comparisons for computer products	Computer products only
Consumer World www.consumerworld.org/ pages/shopping.htm	Consumer information site	Provides product information and price comparison	General consumer and business
eSmarts www.esmarts.com	Consumer-oriented buying site	Provides product and company information and links to suppliers	General consumer, including banks and long-distance phone service
Frictionless Value Comparison Engine www.frictionless.com	Intelligent shopping agent	Has 150 product categories and data on features of 3 million products; takes user through a series of questions to determine the best product for the specified application	General consumer and business
InfoSpace Compare Price vo.infospace.com	Product search	Lists companies that offer a specified product	General consumer and business
Jango www.jango.com	Product search	Has pull-down menus to specify the type of product you want; offers option of looking for product reviews or product prices	General consumer and business
mySimon www.mySimon.com	Comparison shopper	Does price comparisons, tracks prices over time	General consumer, office and computer products
Online Shopping Directory www.samizdat.com/ shopping.html	Directory	Provides a broad directory of sites with links to several hundred shopping, software and reference sites	Consumer, business, government and education products
PriceScan www.pricescan.com	Comparison shopping	Searches Web sites and print ads for prices; has pull-down menus for selecting products	General consumer, office and computer products
R U Sure Shopping Agent www.rusure.com	Price comparison plug-in	Searches supported sites for the best price on an item	General consumer, office and computer products
StreetPrices.com www.streetprices.com	Comparison shopping	Conducts product searches among 250 online merchants; produces a directory of products and prices	General consumer and business
TechBargains www.techbargains.com	Product recommendation	Provides technical articles and product comparisons; will recommend products and explain why, and provides links to suppliers	Computer products
TechShopper www.techweb.com/shopper	Product recommendation	Provides pull-down menus to give product specs and identifies products that meet those specs	Computer products