The search is on for ways to navigate invisible Web sites

Some of the newsiest sites on the Internet are the ones you can’t see.

Conventional search engines never visit a vast area known as “the invisible
Web.” Instead of Hypertext Markup Language pages and eye-catching graphics, this
invisible world consists of databases searchable only by special forms.

Say you routinely check airline schedules for government travel. Much of the flight
information appears on the Web but not on static pages. To set up a flight via the
Internet, you’re forced to visit an airline’s site, enter your request and check
the database to see what’s available. Then you must travel on to the next airline

The government has its own invisible Web. Stop by
and see the dozens of databases listed. Click on any of them and you’ll see a form
that must be filled out before you can find what you want. Search engines do index some of
the databases, but for rapidly changing information, a general-purpose engine isn’t
the best tool. What you need is a universal form that queries multiple Web databases,
saving you the time and trouble of visiting the sites yourself.

To index and navigate the invisible Web, you would send a query that executes in real
time against multiple databases.

Problem is, because we search databases in so many ways, it’s nearly impossible to
create a universal form similar to those used by search engines. What works for an airline
site won’t work for, say, an IRS database of tax information.

The trick is to break databases down into subsets of similar sites and make a limited
form to query just those sites.

Visit for an example. Look
for the entries under Web 4D, then click on the word Auctions.

From this interface, you enter a search term once, then click on the various icons to
search across multiple auction sites.

Say you’re shopping for a fast computer. Simply enter 450 MHz to see what’s

I’d like to see FedWorld and other government sites adopt this approach for
searching government databases. It’s a logical interface and a great time-saver for
databases with similar content and form.

Here are some other navigators for the invisible Web:

Shawn P. McCarthy is a computer journalist, webmaster and Internet programmer for
Cahners Business Information Inc. E-mail him at

Stay Connected

Sign up for our newsletter.

I agree to this site's Privacy Policy.