Lustre to run on ZFS
- By Joab Jackson
- Mar 26, 2008
NEWPORT R.I.'Sun Microsystems developers are working on a version of the Lustre global file system that can run on the company's own next-generation Zetabyte File System (ZFS), said Peter Bojanic, director of Lustre engineering at Sun Microsystems.
Bojanic spoke today at the 23rd annual High Performance Computer and Communications Conference, held this week.
Lustre is an open-source global file system that is heavily used in the supercomputer community. Seven out of the 10 supercomputers at the top of last November's top 500 list
of the world's fastest computers use Lustre, as do 30 percent of the systems in the top 100, Bojanic said. Lustre runs on Sandia National Laboratories' Red Storm Cray supercomputer, as well as with Lawrence Livermore National Laboratory's BlueGene/L system.
A global file system allows very large data sets to be spread across many servers, so they can be accessed more quickly. It relies on a local file system to manage data at the server level. To date, Lustre has been mostly used on the EXT3 file system, which can only index 8T of data per server.
The advantage of using ZFS under Lustre is that ZFS can offer a larger address space, meaning larger sets of data and a larger number of files can be indexed under a single distributed storage system. ZFS has 128-bit addressing
, which means it can index up to 18 million terabytes of data. Lustre has no theoretical limit to the amount of data it can address, Bojanic said.
Besides the ability to index larger sets of data, ZFS will also provide superior file integrity checking capabilities, ensuring that data does not get corrupted. Lustre in turn can assure data doesn't get corrupted as it crosses the network from one server to another.
The major challenge to the porting project now is to improve performance, Bojanic said. The development team's goal is to get the speed of running Lustre over ZFS to within 90 percent of running a highly optimized version of EXT3 alone.
Development of Lustre was formerly overseen by the Cluster File Systems, a company that Sun bought last October. The file system remains under the GNU General Public License.
Sun plans to use Lustre as part of a Linux-based stack for high-performance computing systems. The team is also working on porting Lustre to Sun Solaris and Microsoft Windows.
The development teams hopes to get a version of the ZFS-compatible Lustre released by the end of the year.
Joab Jackson is the senior technology editor for Government Computer News.