DOE completes first cross-country 100-gigabit network

The Energy Department’s Energy Sciences Network and the Internet2 academic research consortium have completed the first transcontinental 100-gigabits/sec network using coherent modulation technology.

The prototype network is being built with a $62 million grant under the American Recovery and Reinvestment Act and is part of DOE’s Advanced Networking Initiative to develop a next-generation science and research infrastructure. ESnet and Internet2 will share capacity on the network, which initially will link DOE supercomputing centers at the Lawrence Berkeley, Argonne and Oak Ridge national laboratories.

The network is intended not only as future-proofing for ESnet bandwidth demands, but also as a testbed for new technology to take full advantage of the 100- gigabits/sec capacity.

Related stories:

Energy building 100-gigabit/sec network to handle data 'explosion'

Energy completes science network upgrade

“One hundred gigabits is evolutionary, not revolutionary,” said ESnet department head Steve Cotter. “What is unique is what we are going to do with it.”

Although 100 gigabit standards already exist, there is more to high-speed networking than pushing bits at a high rate. Science and research networks tend to be used differently than commercial networks, handling a few very large data streams rather than many streams from many applications. Protocols and applications have to be optimized for the higher speeds, and the Energy Department has a program to scale applications now running on 10- gigabits/sec networks up to 100 gigabits. It also is funding development of a 100- gigabits/sec Network Interface Controller to get data on and off the network.

The network is built with the 6500 Packet-Optical Platform that provides coherent optical processing. Coherent modulation of the optical signal allows a signal to travel farther between regeneration.

ESnet provides network connections to 40 sites around the country for 25,000 researchers funded by the DOE Office of Science and has seen its traffic grow by a factor of 10 every 47 months since 1990, Cotter said.

The Energy Department has set a goal of achieving exabit (1,000 petabit) capacity computing by 2020, which would require access to terabit speed networks. The new 100-gigabits/sec network will enable it to meet its expanding bandwidth needs over the next several years, as well as advance toward the terabit goal.

Internet2, established in 1996 to pick up where the National Science Foundation’s NSFnet Internet backbone left off, is a testbed research and education network on which users can try new technologies and applications that might not be feasible on commercial networks. It is owned and operated by a consortium of more than 200 universities, 70 corporations and 45 affiliate members, including government agencies.

Internet2 also provides bigger pipes to accommodate the needs of power users in the research community, such as ESnet.

“We have a long-standing collaboration with Internet2,” Cotter said. “With the 100-gigabit network we have taken the same approach and combined the different stimulus funding packages we have received.”

Each entity will own and operate some of the equipment on the network and each will have access to half of its capacity.

The network will contain instruments to collect data on power consumption, to provide information for the first time on how a high-speed network uses energy. The goal is to enable shifting data and computing loads to make the most efficient use of power sources and to improve equipment design.

“We are hoping that by making this information available we can catalyze research” in equipment design, Cotter said.

Both Internet2 and ESnet also have bought a pair of dark optical fibers from Level 3 Communication’s Tier 1 network that will be used as a testbed for disruptive networking technologies. This type of infrastructure is the holy grail of network research, Cotter said.

“In the past we have been able to offer wavelength” on existing production networks to researchers, he said. But they could not be allowed to try out new risky new protocols or technologies that could disrupt it. The dark fiber will provide a testbed outside the laboratory for experiments.

The prototype network covers nearly 4,000 miles between New York, Washington, Cleveland, Chicago, Kansas City, Denver, Salt Lake City and Sunnyvale. It initially will link the three supercomputer centers and the Manhattan Landing International Exchange Point in New York. It will be expanded to other DOE sites through 2012.

Indiana University also has announced that it will host a high-speed link between the China Education and Research Network and Internet2, as well as other U.S. research and education networks. The link will be the first dedicated connection between the two countries for university research is being established with a National Science Foundation grant.

“This will allow university scientists to move larger sets of data between the U.S. and China more quickly and easily,” said Jimi Williams, director of international networking at the I.U. GlobalNOC.