Los Alamos builds supercomputer network with ultra-fast 3Com Gigabit Ethernet switches

Los Alamos is a scientific institution owned by the U.S. Department of Energy. It will use Avalon for research ranging from astrophysics and global climate modeling to materials and weapons technologies. CNLS, one of the research centers operating within the laboratory, is devoted to the identification and study of fundamental nonlinear phenomena and promote their use in applied research.

The center coordinates the efforts of the Laboratory's premier researchers from many disciplines and many divisions, mobilizing theoreticians and experimentalists in applied math, statistical physics, solid state and materials physics, polymer chemistry, structural biology and biochemistry.

"Avalon sets a new standard for cost-effective supercomputing," said Michael Warren of Los Alamos National Laboratory's Theoretical Astrophysics Group and Avalon's chief architect. "We're able to process huge amounts of data reliably and without latency, even at supercomputer speeds."

Connectivity lies at the core of Avalon's supercomputer design, known as Beowulf architecture. This approach deploys multiple processors, or nodes, that are networked together. Computations are distributed among the nodes via the network, enabling all the processors to work simultaneously on tasks for much quicker results.

Supercomputing with Gigabit Ethernet Performance Avalon is powered by a 3Com network based on a Gigabit Ethernet backbone. The supercomputer's nodes are comprised of 144 DEC desktop computers with 533 megahertz, 64-bit processors and 256 megabytes of RAM. In total, Avalon has nearly 40 gigabytes of RAM. The nodes are all linked at full-duplex 100 megabits-per-second (Mbps) Fast Ethernet speeds to four 3Com 36-port SuperStack II 3900 Gigabit Ethernet switches with three 1000 Mbps Gigabit Ethernet uplink modules. Using their Gigabit Ethernet uplinks, these switches connect to a 12-port SuperStack II 9300 Gigabit Ethernet switch, which serves as the network backbone. This robust design features 12 gigabits of connectivity between the SuperStack II 9300 switch and the SuperStack II 3900 switches.

"One of Avalon's key enabling technologies is the ability of the SuperStack II 3900 switch to trunk its three discrete Gigabit Ethernet uplinks into one," said David Neal, systems administrator for Los Alamos' CNLS and a co-developer of Avalon. "This allows each switch to deliver, in effect, a three gigabit link to the SuperStack II 9300 switch, ensuring the network can sustain extremely high traffic volumes between the 144 nodes."

To reduce costs, Avalon's development team deployed the open source Red Hat Linux 5.1 operating system and other software that are freely available on the Internet.

When Avalon was first built last spring, it featured 70 nodes, but the scalability and high-density of the SuperStack II switches permitted Avalon to attain even greater speeds. "Just half the ports on the SuperStack II 3900 switches had processors linked to them, which meant the network was operating at only half its capacity," added Warren. "The wire-speed performance of the 3900s and the full-duplex 12 gigabit backplane of the 9300 let us expand Avalon to 144 processors, which multiplies its computing power." With 70 nodes, Avalon reached speeds of 19.2 gigaflops, which is 19.2 billion floating point operations per second. At that speed, Avalon was the 315th most powerful computer in the world. With 144 nodes, the supercomputer's performance is expected to approach 60 gigaflops, which will place it within the top one hundred most powerful computing devices. "Avalon began with much more processing power per dollar than any other supercomputer and we improved upon that considerably," said Neal.

Like many supercomputers, Avalon will be used for exotic scientific research, including exploration into the origins of the universe. Avalon, however, demonstrates that Beowulf class supercomputing is accessible to many organizations. "In campus environments, workstations can be linked together and used at night for supercomputing tasks," said Neal. "A high-speed network is essential so the workstations don't compete for bandwidth, undermining performance."

"When we first set up Avalon, we received all the components on a Friday and connected them over the weekend," said Warren. "On Monday, we had one of the most powerful computers on the planet. Supercomputing is no longer super-costly."

Enhancing the Network to Handle Supercomputing

To complement Avalon's enhanced computing performance, CNLS recently upgraded its LAN. Prior to deploying the SuperStack II Switch 9300 switch-based Gigabit Ethernet backbone, CNLS ran its research applications in switched Ethernet/FDDI environment based on a 3Com CoreBuilder 6000 Ethernet/FDDI switch. With the introduction of Avalon and the increasing size and complexity of the programs it ran, this infrastructure created a bottleneck in research operations that routinely include gigabyte file transfers. Dismissing ATM as too costly and FDDI and Fast Ethernet as too slow, CNLS chose Gigabit Ethernet as its backbone technology. With the reliability of 3Com systems already proven in its earlier network, CNLS again turned to 3Com for its upgrade.

Today, the CNLS network consists of a Gigabit Ethernet backbone comprised of a single SuperStack II Switch 9300 switch with 1000 Mbps links to five 36-port SuperStack II Switch 3900 switches and to a single CoreBuilder 3500 Layer 3 high function switch. The CoreBuilder 3500 switch provides layer 3 switching, while the SuperStack II 3900 switches provide full-duplex 100 Mbps Ethernet connections to networked desktops.

Upgraded Los Alamos Local Area Network

The laboratory has also upgraded the LAN for its Center for Nonlinear Studies (CNLS). The 200-user network depends on a Gigabit Ethernet backbone based on a 3Com SuperStack II Switch 9300 Gigabit Ethernet switch. 3Com Fast Ethernet edge devices deliver switched 100Mbps to network desktops. As the conduit for information delivery at CNLS, the high-speed 3Com network creates the stable, high-performance environment needed to support bandwidth-intensive research and analysis activities.

In addition to the hardware, 3Com is providing CNLS with 24 X 7 support, free software upgrades and advance hardware replacement through its comprehensive Guardian (sm) service program. About 3Com Corporation