SC98 - ZIB Berlin linked to NLANR in long-distance supercomputing

The demo, a 3D simulation of Einstein's general relativity theory calculating the collision of neutron stars, will be computed simultaneously on Cray T3E computers at Konrad Zuse Institut in Berlin and the San Diego Supercomputer Center (SDSC). The computations will utilize a parallel computer code called Cactus, developed to solve the complete set of 3D Einstein equations. The two supercomputers will communicate with each other using MPICH-G, a new Globus-enabled implementation of the Message Passing Interface (MPI). MPICH-G incorporates a number of features designed to support efficient execution in wide area environments, including dynamic selection of communication methods and topology information that allows optimized implementations of collective operations. Developed by Argonne National Laboratory, Globus is an integrated set of software components for high-performance networks.

If all goes well, the computations will be done simultaneously on 128 nodes on each of the T3Es in Germany and in San Diego, communicating with each other through a series of international networks, including Deutsche Telekom and DFN-Verein in Germany and Teleglobe, a European international network. The link will then move to CANARIE, a high-speed network in Canada, and will then connect to the National Science Foundation's very high-performance Backbone Network Service (vBNS) through the STAR TAP interconnection point in Chicago. The demo will utilize 128 nodes of the SDSC's 272-node T3E to communicate vertex information (information needed to draw the polygons used in the 3D display) to the conference floor in Orlando via the vBNS. The resulting 3D visualization will be displayed on portable VR systems called ImmersaDesks in the NLANR booth and in the research booths of the National Computational Science Alliance (Alliance), the National Partnership for Advanced Computational Infrastructure (NPACI), the International Grid (I-Grid), and Argonne National Laboratory.

"For me, this is a demonstration of the power of distributed computing," said Jason Novotny, a member of the NLANR distributed applications support team who is working with the international team of scientists who will present the demo. "We are coupling supercomputers and high-speed networks together to handle very powerful simulations. We at NLANR hope to make high-performance distributed computing and access to remote and diverse resources easy and ubiquitous."

Although the research group conducting the demo is no stranger to transatlantic networking experiments, this year's demo is even more ambitious, since it uses computers on two continents to perform the simulation and a third computer in Florida to do the visualization.

"This project requires much more communications bandwidth than before because of the more powerful way we've set things up," said Ed Seidel, the project's senior researcher and a physicist at the Max-Planck-Institut fuer Gravitationsphysik in Potsdam, Germany, as well as the University of Illinois at Urbana-Champaign. "Demonstrations like this pave the way for scientists to exploit high-speed networks to do simulations that a single supercomputer couldn't handle."

At present the network connections are far from perfect--something that the researchers expected in dealing with a still-developing international computational network run by a wide range of private and public entities. John Shalf, of the Alliance's visualization and virtual environments group, has been working with Seidel to finalize plans for the live demo. He said the group had hoped to get speeds of 30 million bits per second (Mbs) throughout its network connections, but will probably get 10 Mbs through some of the network connections in Germany. Even speeds of 10Mbs will be enough to run the intercontinental demo, he added.

The National Laboratory for Applied Network Research is an NSF-supported collaboration to provide technical, engineering and traffic analysis support for NSF's High Performance Connections sites and the broad vBNS user community. NLANR major activities are performed by three teams: a distributed applications support team based at the University of Illinois' National Center for Supercomputing Applications; a measurement and analysis team based at the San Diego Supercomputer Center; and a networking engineering support team based at the Pittsburgh Supercomputing Center.