DFN-Project Gigabit Testbed West experiences discussed at ZAM in Juelich

DFN connects the major academic and research sites throughout Germany with the 155 Mbit/s B-WiN (Research net), others use 34 Mbit/s lines - about 700 institutions and 4000 schools. DFN expects more, e.g. museums or hostels for students. The next step is the Gigabit bandwidth. Therefore DFN started two testbeds, to test the transmission rates, equipment and applications. Testbed West connects the Research Center Juelich and GMD, Sankt Augustin, and in a second step the Universities of Bonn and Cologne and the DLR (German Aerospace Center) near Cologne. The project leadership had ZAM, the testbed was funded with about 9 Mio DM (4.5 Mio EURO) for a two and a half years timeframe. The Gigabit testbed South connects Munich, Garching, Erlangen and Berlin - four times 2.4 Gbit/s.

The project manager Dr. Thomas Eickermann, ZAM, discussed the main goals of the testbed like new communication technologies, transferring the results to the G-WiN, Meta or Grid computing and real applications. It was divided into two phases, from August 1997 to January 2000 the network was installed, tools developed and the applications supported that used the testbed. From 1999 to June 2000, the network was extended, April 1999 the connection from GMD to Academy of Media Arts Cologne, DLR and University of Cologne, 60 km (35 miles) was established. End of 1999 the University of Bonn entered the testbed. All the institutions had a bandwidth of 622 Mbit/s. In August 1997 the first 622 Mbit/s SDH/ATM (Synchronous Digital Hierarchie/Asynchronous Transfer Mode) fibre connection between Juelich and GMD, 120 km (about 75 miles), was opened and upgraded to 2.4 Gbit/s in 1998. An ATM switch ASX 4000 from FORE Systems at both ends distributes the data on the local ATM and HiPPI nets of the centers.

After initial damping problems with the long-range interfaces the connection was very stable. Unsolvable interoperability problems between hard- and software-components of different vendors have been detected, when changing ATM-signaling to hierarchical PNNI. Throughput measurements showed an interesting and surprising fact, the net was no longer the bottleneck but the connected computers. Fast interfaces do not ensure fast processing, the systems cannot process the incoming data - the I/O performance of these computers is not sufficient for such high bandwidths. Peter Wunderling, GMD, mentioned that they discuss the next generation of optical networks, up to 80 Gbit/s and planning Tbit/s. Surely here new testbeds have to be realised.

DFN and the German G-WiN

Based on the experiences of both testbeds, DFN installs a G-WiN (Gigabit Resarch net) in April 2000. The transition to the G-WiN will be supported by the German Ministry of Research (BMBF) with about 80 Million DM (40 Mio EURO). As in the case of B-WiN, the partners and members of DFN pay for the network usage.

The German Telekom and Systemloesungen GmbH (DeTeSystem), Nuernberg, a daughter of German Telekom won the European wide call for tender for the G-WiN. It installs the core network - about 80%. The rest is for other, local or regional carriers who access the core. Cisco donated 10 core net nodes as core routers for free.

In the first step 622 Mbit/s will be realised and extended to 2.4 Gbit/s for level 1 POPS. The core network consists of 29 nodes in two levels, 10 level 1 POPS + 19 regional nodes. The access capacity for the customer in a center will be extended from 622 Mbit/s to 2.5 Gbit/s in 2001 and to 10 Gbit/s in 2003. The backbone capacity between Level 1 POPS will be a multiple of 2.5 Gbit/s in 2001, up to 10 Gbit/s in 2002 and a multiple of 10 Gbit/s in 2003. The Level 2 POPS are directly related to the Level 1, for example Hamburg Level 1, Kiel and Oldenburg Level 2. As DFN is a member organisation, they decided that the usage costs will be independent of the distance. Special services like point to point connections or ATM cost an extra fee. DFN plans a cost structure for IP best effort, starting with 50 000 DM (25 000 EURO) per year for a 2 Mbit/s line and 40 GByte/month and ending with 1.5 Million DM (75 Million EURO) per year for a 622 Mbit/s line and 25 TByte/month. The financing is assured by options, contracts and the requests of the institutions. Depending on the usage and the traffic, the costs can be reduced.

Peter Kaufmann, DFN, discussed future tasks in communications. Topics are IP and GigaEthernet over WDM (Wave-Division Multiplexing), which opens much more bandwidth than 2.5 Gbit/s. Other issues are the test of Terabit routers and the development of management and monitoring tools for DFN and its customers.

Applications

Some eight applications have been presented, multimedia applications, multiscale molecule dynamics, visualisation at the responsive workbench e.g. in medicine, distributed computation of climate and weather models, distributed, virtual production of television, distributed traffic simulations and visualisation, magneto-encephalography and pollution propagation.

Climate and Weather Modelling

GMD and Alfred-Wegener-Institute for Polar and Ocean Research, Bremerhaven, coupled an Ocean-Ice-Model (MOM-2) on the T3E with an atmosphere model (IFS) on the SP2. After each simulation step temperature, wind and the rain-fall data at the ocean surface are exchanged. The parallelisation of IFS is MPI-based, while MOM uses the T3E shmem-library. The external coupling of both programmes was realised by MetaMPI, a Metacomputing-MPI library developed by Pallas GmbH. The coupling of the codes introduced a communication overhead of about 10% due to a latency of 10 msec between the machines compared to a 10 microsec for internal communication.

MEG Activities in the Human Brain

Brain research is one of the main activities of the Institute of Medicine at Research Center Juelich. Neural processes produce electrical currents with high density. The magnetic field can be measured outside the head using Magneto-encephalography (MEG) - a method that puts no strain on the test person or patient.

These data give hints on the current sources in the brain. The magnetic field of the brain is 10 ** - 9 Tesla - 100,000 times weaker than the magnetic field of the earth with 10 ** - 4 Tesla. The timely resolution is better than one millisecond. One weakness of MEG is the low spatial resolution. Out of 148 measurement points - an underdetermined system - only approximate current distributions can be estimated. A well established method is the so-called MUSIC (MUltiple SIgnal Classification) algorithm.

Due to the nature of this algorithm, optimal overall performance is achieved when parts are implemented on a massively parallel supercomputer - a Cray T3E and other parts on a vector supercomputer - a Cray T90. Therefore, ZAM implemented a distributed parallel MUSIC-programme based on the MetaMPI library. In a simple example - the neural response to a click-noise applied to the left ear, the distributed version was twice as fast as a stand-alone version. Thus it becomes possible to analyse a 10 minute experiment within 20 instead of 40 minutes, a major improvement when the programme is used for quality control or optimisation of stimulation conditions during MEG-experiments. A possible future application is the development of a "brain pacemaker" for Parkinson's disease.

The Web site with more information can be found at: www.fz-juelich.de/gigabit

For detailed questions, Dr. Thomas Eickermann (ZAM, Research Center Juelich), the Project Leader of Gigabit Testbed West, is available at: Th.Eickermann@fz-juelich.de