Batch processing

In this age of 24/7 real-time processing, we tend to think of batch processing as some legacy from an earlier pre-networked era. At the JavaOne conference last week, SpringSource architect David Syer offered some pretty good reasons as to why batch processing is still with us, and will probably always be around.

Some duties are most efficiently executed when bundled together, especially repetitive ones that can be done during times of inactivity, Syer argued. He offered a few well-chosen analogies: Mass transit is a good example of batch processing. People wait at a bus stop, and a bus appears and collects all of them at once. That approach is far more efficient, resource-wise, than each person driving a car to his or her destination.

Grocery shopping is another batch process. When you shop, you don't purchase items one at a time. You don't go into a store, pick up an orange, go through checkout to purchase the said orange, take it back to the car, and then go back into the store to purchase an apple, take that back to the car, and so on. You collect all the items you want to buy and purchase them in one transaction: Batch processing.

In his presentation, Syer went over some of the trickier problems that can plague batch operations. For instance, say you create an operation that processes 100 items before writing the results to a database. Such "chunking" as he called it, optimizes the use of the database and the network.

But say that when you send one batch, a single item'you are not sure which one'causes the whole process of writing the results to the database to fail. How do you know which item was the troublesome one? If your operation tries to flush the whole chunk again, the same thing will happen.

For this case, Syer suggested, you create a process that kicks in when such an exception occurs. This process writes each item to the database individually.

Sun Microsystems usually posts Webcasts of the JavaOne talks a few weeks after the conference, for those grappling with batch processing, Syer's talk may be worth checking out.

Posted by Joab Jackson on May 13, 2008 at 2:33 PM