Energy building 100-gigabit/sec network to handle data 'explosion'
Connecting state and local government leaders
The goal of the Advanced Networking Initiative, being built in conjunction with Internet2 with Recovery Act funding, is to develop a 1 terabit/sec science and research network.
The Lawrence Berkeley National Laboratory and the Internet2 advanced networking consortium announced July 13 plans to build a prototype 100-gigabits/sec prototype network as the first step toward an eventual terabit-speed network.
The prototype network will be built for the Energy Department’s Energy Sciences Network (ESnet) with a $62 million grant under the economic stimulus law. It is part of the Advanced Networking Initiative to produce a next-generation science and research infrastructure, and will include research on energy efficient networking.
“We are seeing now the globalization of science,” said Rob Vietzke, Internet2 executive director of network services. Projects such as Europe’s Large Hadron Collider produce large volumes of data that are shared and analyzed by scientists around the world, and the volume of research network traffic is growing twice as fast as commercial Internet traffic. “We have to be ahead of the demand curve in commercial offerings.”
Related coverage:
Energy gets jump on implementing DNS security on ESnet research network
Underground lab could help address modern science queries
“We are seeing an explosion of data,” said Steve Cotter, ESnet department head. “Historically our traffic has grown by a factor of 10 every 47 months, since 1990.”
Energy has set a goal of achieving exabit (1,000 petabits) capacity computing by 2020, which would require access to terabit speed networks. Cotter said that, given the time it has taken to get to 100-gigabit networking speeds since 10-gigabit networking was introduced in the late 1990s, “we are going to be hard pressed to get to terabits by 2020.”
The ANI prototype will be one of the first national-scale deployments of 100-gigabits/sec networking, and initially will connect three of Energy's unclassified supercomputing centers: The National Energy Research Scientific Computing Center at Berkeley Lab, the Oak Ridge Leadership Computing Facility in Tennessee, and the Argonne Leadership Computing Facility in Illinois. It also will connect with the Manhattan Landing International Exchange Point in New York.
Internet2, established in 1996 to pick up where the National Science Foundation’s NSFnet Internet backbone left off, is a testbed research and education network on which users can try new technologies and applications that might not be feasible on commercial networks. Internet2 also provides bigger pipes to accommodate the needs of power users in the research community, such as ESnet. A consortium of more than 200 universities, 70 corporations and 45 affiliate members, including government agencies, operate the network.
ESnet provides high-bandwidth connections linking scientists at more than 40 DOE laboratories, universities and other research institutions. Most of its links now are operating at 20 gigabits/sec to 40 gigabits/sec, capacities that are achieved by bundling 10-gigabits/sec optical links.
ESnet and Internet2 helped to pioneer standards for 100-gigabits/sec networking.
“The standards are out and they are interoperable for 100 gigabit,” Vietzke said. But broader vendor adoption is needed, along with additional features, interfaces and applications. The ANI prototype will provide a breakable testbed on which disruptive new technologies can be tried and perfected.
The ANI prototype network will be built using dark fiber on Level 3 Communications’ Tier 1 fiber-optic network. ESnet will have an option to access to 4.4 terabits/sec capacity on the fiber out of a total of 8.8 terabits that Internet2 is using to build out the 100-gigabits/sec U.S. Unified Community Anchor Network. Additional fiber-optic cable will be laid in the San Francisco Bay area and Chicago area to support the ANI network. The prototype network will use Ciena’s 6500 packet-optical platform.
Although 100-gigabit standards already exist, there is more to high-speed networking than pushing bits at a high rate. Science and research networks tend to be used differently than commercial networks, handling a few very large data streams rather than many streams from many applications. Protocols and applications have to be optimized for the higher speeds, and DOE has a program to scale applications now running on 10-gigabits/sec networks up to 100 gigabits/sec.
Energy's $30 million Magellan program is examining cloud computing as a model for high end science with supercomputers, and has found that a well-tuned platform such as Magellan can have a significant improvement in performance over a traditional commercial service.
“Security at 100 gigabits is going to be a significant challenge,” Cotter said. Energy is funding development of a 100-gigabits/sec Network Interface Controller to help.
The ANI also will advance green computing, he said. “We are going to instrument it for power consumption,” gathering data from equipment at all layers of the network on a national scale. “Surprisingly little research has been done on power consumption on large-scale networks.”
IT now consumes as much energy as the airline industry and demand is growing much faster, Cotter said. “If we don’t change how we design networking equipment, which has been designed to maximize capacity, to also maximize efficiency, there won’t be enough power to run all the networking equipment.”