Vendor unveils streaming database
Connecting state and local government leaders
StreamBase Systems Inc. has released a new type of database that eliminates the need to index and commit data to storage before it can be queried.
StreamBase Systems Inc. has released a new type of database that eliminates the need to index and commit data to storage before it can be queried, cutting the time it takes to process the data.
The software, called a Stream Processing Engine, handles 'high volumes of data that are coming in very quickly,' said Barry Morris, president and chief executive officer of the Lexington, Mass.-based company.
Although the company first marketed beta versions of the product to the financial industry, it is now showing the technology to the government market, particularly to agencies testing sensor networks, Morris said. StreamBase formally released the software last week.
Like many traditional transaction-based relational databases, the software stores data entirely in the RAM of a server or servers, eliminating data calls to slower hard drives. Unlike those database-driven designs, however, StreamBase's software does not need to index incoming material into a table before it can be queried, he said.
As a result, agencies can use the software in situations where it is necessary to get a response time 'in milliseconds,' Morris said.
The Homeland Security Department could use the software to monitor multiple streams of sensory data, such as those generated from radio frequency identification tags or other small sensors. The Defense Department could deploy the software to monitor sensor data from vehicles, soldiers' uniforms and other resources, showing the positions and conditions of the troops and equipment.
'A general might want to know what the state of readiness is, how many soldiers has he lost, how much fuel is left,' Morris said.
Although some organizations have custom-built this sort of functionality, StreamBase's release is the first commercial product of this sort, Morris said. The software's architect, Mike Stonebraker, chief technical officer for StreamBase, was formerly the main architect of the Ingres relational database, now offered by Computer Associates International Inc. He also designed the open-source PostGres database when he was a professor at the University of California at Berkeley.
The processing engine maintains a stream of data messages, each formatted as either a number, string or time-stamp. The company has demonstrated that the Stream Processing Engine can manage over 140,000 messages per second. Administrators can create queries based on time constraints, on when a certain number of messages have been received, or other factors. The company also offers adapters that will export or import data from messaging interfaces of other applications.
StreamBase licenses the Stream Processing Engine on a per-processor basis, with a typical deployment costing about $60,000 per year. The software runs on most Unix and Linux platforms. Later this month, the company will offer a version of the software for computers running Microsoft Windows. An average implementation can run smoothly on a single desktop computer with as little as 512M of memory. The software also has failover and a check-pointing mechanism to recover from hardware failures, Morris said.
Tom Rizzo, Microsoft's director of product management for SQL Server, said that while StreamBase's software may be good for niche applications, SQL Server has many features that speed database querying times, making ingestion speed less of an issue in many cases. For instance, SQL Server moves frequently read data to a cache in working memory. It also features an automated process of optimizing queries to reduce response time.
Rizzo also noted that it might be difficult to run larger databases'those ranging over multiple terabytes'only in working memory.
'Eventually you are going to page [your data] out to disk,' he said.
NEXT STORY: Army hires CSC for logistics project