GCN Tech Blog

By GCN Staff

Blog archive

Batch processing

In this age of 24/7 real-time processing, we tend to think of batch processing as some legacy from an earlier pre-networked era. At the JavaOne conference last week, SpringSource architect David Syer offered some pretty good reasons as to why batch processing is still with us, and will probably always be around.

Some duties are most efficiently executed when bundled together, especially repetitive ones that can be done during times of inactivity, Syer argued. He offered a few well-chosen analogies: Mass transit is a good example of batch processing. People wait at a bus stop, and a bus appears and collects all of them at once. That approach is far more efficient, resource-wise, than each person driving a car to his or her destination.

Grocery shopping is another batch process. When you shop, you don't purchase items one at a time. You don't go into a store, pick up an orange, go through checkout to purchase the said orange, take it back to the car, and then go back into the store to purchase an apple, take that back to the car, and so on. You collect all the items you want to buy and purchase them in one transaction: Batch processing.

In his presentation, Syer went over some of the trickier problems that can plague batch operations. For instance, say you create an operation that processes 100 items before writing the results to a database. Such "chunking" as he called it, optimizes the use of the database and the network.

But say that when you send one batch, a single item'you are not sure which one'causes the whole process of writing the results to the database to fail. How do you know which item was the troublesome one? If your operation tries to flush the whole chunk again, the same thing will happen.

For this case, Syer suggested, you create a process that kicks in when such an exception occurs. This process writes each item to the database individually.

Sun Microsystems usually posts Webcasts of the JavaOne talks a few weeks after the conference, for those grappling with batch processing, Syer's talk may be worth checking out.

Posted by Joab Jackson on May 13, 2008 at 9:39 AM


  • Telecommunications
    Stock photo ID: 658810513 By asharkyu

    GSA extends EIS deadline to 2023

    Agencies are getting up to three more years on existing telecom contracts before having to shift to the $50 billion Enterprise Infrastructure Solutions vehicle.

  • Workforce
    Shutterstock image ID: 569172169 By Zenzen

    OMB looks to retrain feds to fill cyber needs

    The federal government is taking steps to fill high-demand, skills-gap positions in tech by retraining employees already working within agencies without a cyber or IT background.

  • Acquisition
    GSA Headquarters (Photo by Rena Schild/Shutterstock)

    GSA to consolidate multiple award schedules

    The General Services Administration plans to consolidate dozens of its buying schedules across product areas including IT and services to reduce duplication.

Stay Connected

Sign up for our newsletter.

I agree to this site's Privacy Policy.