Large Hadron Collider Photo by Maximilien Brice/CERN

Internet2 backbone serving national supercomputer network

A national network of university supercomputers has migrated to the Internet2 8.8 terabit-per-second optical backbone, which offers advanced networking services as well as additional bandwidth for its high performance computing projects.

The Extreme Science and Engineering Discovery Environment (XSEDE) includes 17 universities and research facilities hosting high-performance computing and throughput machines, and advanced visualization, storage and gateway systems supporting more than 8,000 scientists.

The size of the pipes is important in enabling their research, said XSEDE project director John Towns of the University of Illinois. “Simple bandwidth availability will make a difference,” Towns said. “We’re already seeing that.”

But the task of moving increasingly large files between supercomputers requires new networking tools as well as big pipes. That is one of the reasons XSEDE chose Internet2 as its backbone provider, Towns said. “They were interested in partnering with us to develop the network services that our users need.”

Internet2 is a member-owned organization with more than 200 participating universities, labs and companies, providing advanced networking services to educational and research institutions. Last summer it finished building out a 15,000-mile core network offering 8.8 terabits/sec in 88 bundled 100-gigabit channels.

Internet2 network of advanced service nodes

XSEDE is an umbrella organization funded by a five-year, $121 million National Science Foundation grant to provide an enhanced environment for high-performance computing that is becoming more critical to “big science” projects. Projects require not only the computing power to do complex simulations and analytics, but also involve large data sets both as input and output.

XSEDE members

XSEDE is led by the University of Illinois' National Center for Supercomputing Applications. Members include:

  • Cornell University Center for Advanced Computing
  • Indiana University
  • Jülich Supercomputing Centre
  • National Center for Atmospheric Research
  • National Center for Supercomputing Applications - University of Illinois at Urbana-Champaign
  • National Institute for Computational Sciences - University of Tennessee Knoxville/Oak Ridge National Laboratory
  • Ohio Supercomputer Center - The Ohio State University
  • Pittsburgh Supercomputing Center - Carnegie Mellon University/University of Pittsburgh
  • Purdue University
  • Rice University
  • San Diego Supercomputer Center - University of California San Diego
  • Shodor Education Foundation
  • Southeastern Universities Research Association
  • Texas Advanced Computing Center - The University of Texas at Austin
  • University of California Berkeley
  • University of Chicago
  • University of Virginia
  • With an upfront cost of $30 million or more for a supercomputer, plus $1 to $2 million a year for power as well as dedicated environment and staff, these systems are hosted in only a handful of institutions, Towns said. Each has its own strengths and characteristics, and researchers often need to use separate systems to prepare data for input, run simulations or modeling, and analyze the results. In the past, data often was moved from one facility to another by hand. High-speed testbed and research networks now provide a way to move the data more efficiently.

    XSEDE had a backbone contract with National LambdaRail, a university-owned high-speed national research and education network whose users include U.S. laboratories participating in Large Hadron Collider projects. When that contract expired it moved to Internet2 to take advantage of research and development of services supporting bandwidth-intensive applications, such as the XSEDE-wide File System for handling large files.

    Commercial carriers are deploying 100-gig networks, but “the way Internet2 deploys 100G is different,” said Rob Vietzke, Internet2’s vice president of network services. The bandwidth is available to handle the high-performance needs of a relatively few users rather than for aggregating a large number of small users. This means there are tens of gigabits of throughput available for bursts of traffic.

    The 17-member XSEDE institutions link to Internet2 through the nearest of the network’s 21 Advanced Layer 2 Service nodes. Most have 10G connections, although two schools, Indiana University and Purdue University, have 100G connections.

    “It’s just a matter of time and economics,” Vietzke said of the larger connections for the two Hoosier schools. “Indiana was an early investor in 100G. They were in the forefront.”


    • Records management: Look beyond the NARA mandates

      Pandemic tests electronic records management

      Between the rush enable more virtual collaboration, stalled digitization of archived records and managing records that reside in datasets, records management executives are sorting through new challenges.

    • boy learning at home (Travelpixs/

      Tucson’s community wireless bridges the digital divide

      The city built cell sites at government-owned facilities such as fire departments and libraries that were already connected to Tucson’s existing fiber backbone.

    Stay Connected