High Performance Computing in Poland and IBM SP2 systems

Krakow, 20-10 -1996 In the paper computer resources in Poland and their applications in research in different fields of High Performance Computing are presented with special interest given to IBM SP2 systems.

For the last decades the computational science has become equally important to the theoretical and experimental sciences. The essential goal of developing new computer architectures and new numerical algorithms is to run larger and more complicated applications faster over time. For program developing at least three elements should be taken into account by the user: the problem in hand, the available architecture and the algorithm together with programming tools. High performance computing (HPC) is usually the only choice for problems which classically belong to Grand Challenges.

Several examples could be: climate modelling, fluid turbulence, pollution dispersion, human genome, ocean circulation, quantum chromodynamics, semi- and superconductor modelling, vision and cognition.

Metacomputing, one of the most exciting ideas in computational science at present, is believed to empowers users by providing transparent access to a variety of services, like

computing services,
graphical, multimedia and scientific visualisation services,
storage (including hierarchical storage management) services.

The network metacomputer should make large scale computing more efficient and cost-effective, with special interest given to the Grand Challenges problems. Metacomputing is closely related to high-bandwidth network protocols. Each scientific community should possess access to the metacomputing resources.

High Performance computing in Poland

High performance computing (HPC) is one of the most fundamental tools in science and engineering. Since Polish science is an important element of the universal science HPC is present in all aspects in research and application development in Poland.

HPC systems in Poland come from almost all major computer vendors and represent different computer architectures. There are multiprocessor vector machines from Cray Research and Convex, parallel computers with shared address space memory and bus-based processor interconnections (SGI Power Challenge/Onyx), parallel computers with distributed memory, message-passing architecture and multistage interconnection network (IBM SP2) as well as parallel computers with virtual-shared distributed memory and multilevel organisation of processor interconnections (HP/Convex Exemplar SPP). Applications of data-parallel computing model and message-passing paradigm are possible. Some of the centres running these powerful machines are equipped with hierarchical mass storage systems and efficient graphics processors.

The HPC systems are provided with sophisticated application software packages and programming tools. They are in principle available to all scientists in Poland. Diversity of programming resources makes computing of different kinds possible.

Case study: Development of HPC in Cracow

In early 1970's a group of scientists recognised that computers and computing were important tools to keep up polish research with universal science achievements. Due to their efforts two regional computer centres were established: in Cracow and in Swierk at the Institute of Nuclear Research.

The Academic Computer Centre CYFRONET - Kraków (ACC CYFRONET) was established in 1973 as an independent non-profit organisation under the authority of the Ministry of Science, High Education and Technology (at present the Ministry of National Education) to provide Cracow academic community and research institute with power facilities for tasks which were beyond the scope of their own computer resources.

There are three main tasks for ACC CYFRONET activity

delivering power combined with a wide range of application software to the members of the academic and research staff and students of more than 10 universities and research institutes of Cracow scientific community as well as to the users from other polish scientific centres via wide area networks. ACC CYFRONET - Kraków services the specialists of practically all disciplines: mathematicians, physicists, physicians, sociologists, chemists, astronomers, engineers, linguists, biologists and others.
delivering network and information services - the capability to access world-wide Internet resources by scientists, students and many others, like high school pupils and commercial institutions from area of Cracow and Southern Poland.
scientific and educational activity concerning problems of high performance computing, applications of the computer networks and information science. ACC CYFRONET - Kraków actively organises international and national scientific conferences, seminars and other activities like training that popularise new solutions in the world of computing and networking.

Originally ACC CYFRONET - Kraków was fitted out with a CDC Cyber 72 computer, which for many years was the basic computer facility. The fundamental decision to update the resources was taken in 1989 to introduce a vector machine for Cracow Community. After two years of attempts to obtain US export license a Convex C120 single-processor system was installed. It was the first vector computer in Eastern Europe. The first machine was followed a year later by a C3210, a more advanced model which utilisation by scientists achieved soon 90%. This fact defined the need for further development which resulted in two-processor C3820 machine in 1993.

For the last years utilisation of both vector machines: C3220 (upgraded from C3210) and C3820 has been stabilised again at 90% level. Such a high level of utilisation induced Scientific Committee of ACC CYFRONET - Kraków to apply for additional vector power which will result in 1996 in upgrade C3820 to four-processor C3840 machine.

During last years due to application and development of new parallel computer architectures and algorithms needs for parallel computing in Cracow Scientific Community have become evident. After careful analysis of architecture details and verification of the competitive offers from a number of vendors by a special commission the HP/Convex Exemplar SPP1000/XA with 16 processors was chosen in 1994, upgraded to model SPP1600 with 32 processors in 1995. Further development for 1996 considers upgrade to SPP1600 model and installation of 5-node IBM SP2.

Cracow metropolitan area network has been built mainly using the FDDI technology (about 50 km ring) but we realised that in the future it should be changed to the modern local-area networking with ATM technology. In ACC CYFRONET - Kraków the CISCO routers are connected using ATM; this technology is also applied in connecting some university computer centres with ACC CYFRONET resources.

Other regional computer centres in Poland were established subsequently and followed to some extend organisation and purposes of ACC CYFRONET - Kraków.

Computing resources

Polish computing resources are characterised on the background of TOP500 list [1]. They are estimated on the information published in [2] by the representatives from the centres, on the public domain list of supercomputing equipment in Poland [3] and on local WWW pages. Computers with peak performance greater or equal to 0.2 Gflop/s are only taken into considerations.

Number of systems installed Installed Rpeak [Gflop/s] (aggregated)
Poland World Total Poland World Total
SGI 12 125 10 715
IBM 4 116 13 1297
Cray 5 92 7 1585
Intel --- 34 --- 719
Fujitsu --- 33 --- 943
TMC --- 31 --- 804
HP/Convex 2 20 8 158
NEC --- 20 --- 297
Parsytec --- 10 --- 95
others --- 19 --- 292
Total 23 500 38 6904

Total number of systems installed and peak-performance based on TOP500 statistics and estimated for Poland

*Total number of systems installed and peak-performance based on TOP500 statistics and estimated for Poland*
	Number of systems installed	Installed Rpeak [Gflop/s] (aggregated)
	Poland	World Total	Poland	World Total
SGI	12	125	10	715
IBM	4	116	13	1297
Cray	5	92	7	1585
Intel	---	34	---	719
Fujitsu	---	33	---	943
TMC	---	31	---	804
HP/Convex	2	20	8	158
NEC	---	20	---	297
Parsytec	---	10	---	95
others	---	19	---	292
Total	23	500	38	6904

In the table, the number of vector and parallel systems installed in Poland and their peak-performance is presented. Most of them come from SGI, for which Power Challenge/Onyx dominates the market. The number of processors in a single configuration is rather moderate (2-12 MIPS R8000), well below the machine limit. There are also some installations based on 2-processor Challenge computers (with performance less than 0.2 Gflop/s) thus not included in comparison.

The next vendor represented well in Poland is Cray Research; the machines are mainly multiprocessor vector computers (models Y-MP EL, 4E and J916), one case concerns parallel computer CS6400 based on SPARC processors. Machines from HP/Convex are vector and parallel systems with C3820 as a vector number cruncher and parallel Exemplar SPP1200/XA (with 32 PA7200 processors). Other systems with low performance (like C3220) are used for program development.

Although the number of IBM SP2 systems installed in Poland is rather small they deliver most of the computing power. Since the power is concentrated in four centres only and not distributed across the country it is more convenient to use. The systems are equipped with small number of processing nodes (3 installations with 15 Power2 processors and 1 site with 5 processors) in comparison with the most powerful installations in the world (cf. [1]). The systems can easily be upgraded without existing software penalty to hundreds of processors due to scalability features of the Vulcan switch, with constitutes interconnection layer between processors.

Overview of Polish HPC centres

According to the decision taken by the Committee for Scientific Research [4] the main computing resources in Poland are located in centres organised in Warsaw, Poznan, Gdansk and Cracow. Later on Wroclaw centre has been included.

The most powerful vector computers are installed in Poznan Supercomputer-Network Centre (PCSS) and in Interdisciplinary Centre for Mathematical and Computer Modelling (ICMMK) in Warsaw. Although they are efficiently used, more computational power is delivered from parallel machines. At present HP/Convex Exemplar SPP1200 with 32 nodes is the most powerful computer of this kind. Since parallel machines are relatively easy and cost-effective in upgrades this shows possibility for incremental increase of computing power in Poland. It is worth to mention, that the most powerful worldwide installations are based on Hitachi and Fujitsu parallel systems and on Intel Paragon (which production is cancelled), on Cray T3D/T3E and on IBM SP2 with hundreds of processors. Typical examples of SP2 systems concern Cornell, Poughkeepsie and Maui high performance computer centres which shows, that having in Poland some base IBM SP2 configurations further development is profitable.

Aggregated peak performance, R peak [Gflop/s]
Site System Number
of processors Rpeak TOP500
position
1. PCSS Poznan IBM SP2 15 4.0 ---
Cray J916 16 3.3 ---
SGI PC XL 12 4.3 490
Cray Y-MP EL 4 0.5 ---
Total 12.1
2. ACC CYFRONET HP Exemplar SPP1200 32 7.7 406
Kraków IBM SP2 5 1.3 ---
HP/Convex C3820 2 0.5 ---
Total 9.5
3. TASK Gdansk IBM SP2 15 4.0 ---
SGI PC XL 4 1.2 ---
Total 5.2
4. WCSS Wroclaw IBM SP2 15 4.0 ---
Total 4.0
5. ICMMK Warsaw Cray Y-MP 4E 4 1.3 ---
Cray Y-MP EL98 8 1.1 ---
Total 2.4

Aggregated peak performance of regional centres (TOP 500 Supercomputer Sites, June 7, 1996)}

**Aggregated peak performance of regional centres (TOP 500 Supercomputer Sites, June 7, 1996)}**
		Aggregated peak performance, R peak [Gflop/s]
	Site	System	Number of processors	Rpeak	TOP500 position
1.	PCSS Poznan	IBM SP2	15	4.0	---
		Cray J916	16	3.3	---
		SGI PC XL	12	4.3	490
		Cray Y-MP EL	4	0.5	---
	Total			12.1
2.	ACC CYFRONET	HP Exemplar SPP1200	32	7.7	406
	Kraków	IBM SP2	5	1.3	---
		HP/Convex C3820	2	0.5	---
	Total			9.5
3.	TASK Gdansk	IBM SP2	15	4.0	---
		SGI PC XL	4	1.2	---
	Total			5.2
4.	WCSS Wroclaw	IBM SP2	15	4.0	---
	Total			4.0
5.	ICMMK Warsaw	Cray Y-MP 4E	4	1.3	---
		Cray Y-MP EL98	8	1.1	---
	Total			2.4

Due to high level of utilisation the Centres actively strain after resources development. For example in ACC CYFRONET - Kraków the upgrades of C3820 and of SPP1200 are planned for 1996.

Polish scientific community is supported with tools and programming environments for developing user applications. In most centres sophisticated scientific and engineering packages are also available. Perhaps the best example are ICMMK (Warsaw) and ACC CYFRONET - Kraków software resources [5, 6], which offer the community diversity of packages. The following packages are available: molecular modelling (Biosym, Sybyl-Tripos, Unichem), quantum chemistry (QCMD, Mopac, GAMESS, dGauss, Gaussian94), molecular dynamics (Amber, Argos, Gromos, UHBD, QCMD), general purpose (Maple, Matlab, ACSL), CAD/CAE systems (ABAQUS, EMAS, Nastran, NISA/EMAG, BEASY, Pro/Mechanica, ADAMS), information data bases, information systems and data processing (Beilstein, MDL, Unity, CCSD, SAS), numerical libraries (Math Advantage, NAG library, CernLibrary), visualization systems (AVS, Iris Explorer, Khoros,Vis5D).

Overview of research activity

Polish science is an important element of the universal science. Scientists from different polish centres actively take part in international co-operation and grants, thus efficient network and computer resources together with programming packages make the activity feasible.

Almost all fields of science are represented in Poland. They concern theoretical fundamental research and applications studies in Grand Challenges problems. Below some of them are shortly presented.

Warsaw Scientific Community

In Warsaw Scientific Community high performance computing is concentrated in ICMMK at Warsaw University and concerns mainly [5]

computational science and mathematical modelling in chemistry, physics, biology, and material science
large scale computing and scientific visualisation

Some important fields refer to molecular biology with simulation research of enzyme reactions, protein properties, molecular recognition processes and protein folding as well as to quantum chromodynamics, molecular dynamics and phase transitions. Neural networks are used for NMR spectra analysis and analysis of trajectories of biomolecular systems. Other examples cover computer graphics and artificial intelligence methods.

Cracow Scientific Community

In Cracow the most of the computer resources are used by the scientists from Jagiellonian University (together with Collegium Medicum), University of Mining and Metallurgy and Instutite of Nuclear Physics [7].

In 1995/96 the most time consuming applications were realized in the field of astrophysics, other areas cover investigations of heavy ions reactions, high energy physics, protein structures, phase transitions, QCD. Researchers from University of Mining and Metallurgy are concentrated more on application computing - pattern recognition, parallel linear algebra algorithms, development of efficient algorithms for molecular dynamics and lattice gas, casting, reliability of electrical machines.

The scientists from Cracow Community actively took part in the last three subsequent International Conferences on High Performance Computing and Networking, presenting 12 papers [8]-[19].

Research in Poznan Scientific Community

Scientific research in Poznan Community is concentred in PCSS [20]. It covers development of vector, parallel and distributed algorithms for scientific and technology applications (i.e. for molecular and quantum physics and chemistry, mechanics, crystallography, biomedical sciences and computer science) and investigations in the field of integrated system and network management. PCSS, Poznan University, Poznan Technical University and institutes of Polish Academy of Sciences cooperate in the fields of atomic and molecular systems (with Monte Carlo, molecular dynamics and Brownian dynamics) for RNA and other classical and quantum systems modelling.

Other activities concern biomolecular problems, organic chemistry, theoretical physics, expert systems and pattern recognition for medical diagnosis.

Research in Wroclaw Scientific Community

Scientific research in Wroclaw region is coordinated by Wroclaw Network-Supercomputer Center (WCSS) [21]. Main areas refer to fundamental and applications problems for quantum chemistry, molecular biology and physics. Development of new numerical algorithms and programs consider molecular dynamics and Monte Carlo methods, lattice gas and finite elements algorithms. The research is mainly done on IBM SP2 platform with GAUSSIAN94, GAMESS, DeFT and MOPAC packages.

Conclusions

As a result of scientific and financial policy of the Committee for Scientific Research five powerful regional HPC centers have been established and supported with modern hardware and software resources. They play an important role in polish science and in international cooperation.

Past and current experiences show that this way of HPC introducing in Poland is cost effective and highly profitable for research performed in different regions of the country. Such a policy seems to be a better solution than concentration of resources in one national centre. For keeping polish science at high level it is necessary to continue the policy unchanged and still developing the existing regional centres.

Jacek Kitowski

This article was reprinted with permission from the Conference proceedings of the Sup'Eur '96 Conference in Kraków, September 8-11, 1996, ISBN 83-902363-2-5. edited by Marian Bubak and Jazek Moscinski.