ISC2003 - Earth Simulator Delivers New Science Results

What became clear is that the Earth Simulator?s importance is not as much the installation of a single powerful machine, but as an investment in science driven computing. The Japanese scientific community defined its needs, looked at the technology available and set about to built the best system for their scientific requirements.

The Earth Simulator is giving Japanese scientific communities a material advantage and making them more attractive as international collaborators. The Earth Simulator should not be seen as a special purpose machine. Peak performance does not reveal the real impact of the Earth Simulator. Japanese scientific policy is to build strategic partnerships in climate, nano-science and fusion, dominating simulation in many disciplines, not just climate modeling.

As Horst Simon from NERSC said: ?To optimize architectures for scientific computing, it is necessary to establish feedback between scientific applications and computer design over multiple generations of machines. The Japanese Earth Simulator implemented one cycle of this feedback, and made dramatic progress?.

The keynote speaker Professor Dr. Tetsuya Sato, Director General of the Earth Simulator, gave a brief overview of the Earth Simulator project from its beginnings in 1997 to its completion at the end of February 2002. Prior to his current position, Dr. Sato held many academic posts and has a number of awards, including the prestigious JIP Excellence Award of Nikkei Science and the Nishina Memorial Award.

To remind readers, both the physical characteristics and the electronics employed are breathtaking. This vector massively parallel, 5120 processor system with an 8Gflop/s processor delivered on a 2x2cms chip, rated at 40Tflop/s peak, uses over 3,000 Km of electric cables to connect it together.

As one measure of its impact, the last sixteen months over six thousand guests visited the Earth Simulator, 550 of these were from abroad. The number of Website accesses, are mostly from abroad with only 10% from Japan. Interviews from news media also show that the E.S. is making more waves abroad than in Japan.

A Committee of distinguished scientists from various fields is in place, promoting the Earth Simulator. It selects and authorizes priority scientific fields, computer resources allocation, participates in annual and medium-range planning and provides guidelines for proposal selection. The resource committee has allocated 35% for climate science, 20% for solid earth science, 10% for computer science, 15% for innovative projects from other fields and 20% left, for the director?s discretion. The Earth Simulator has been operational for a year, with the first set of projects approved by the project selection committee in June 2002.

International collaboration includes, the Hadley Climate Center, UK, CIRA, Italy, SCRIPPS Institute of Oceanography, USA, the Canadian Met Office, and CNRS/IFREMER, France. NERSC in USA, DKRZ and some other institutes in Germany, negotiations are in progress. Those who wish to collaborate should contact Dr. Sato, to discuss the mechanics. Email: tetsuya@es.jamstec.go.jp

The Earth Simulator is almost 100% utilized and has been running at about 35% efficiency over the whole of last year.

One of the primary applications for the Earth Simulator is the high resolution Atmospheric General Circulation Model (AGCM). With its 40Tflop/s performance plus 10Tbytes of memory and 1Tbytes communications switch the Earth Simulator can use a 10Km resolution mesh and simulate regional phenomena such as typhoons / hurricanes / cyclones or El Nino / La Nina completely. At the same time, the same model can be used to analyze long-range phenomena such as global warming.

Before the Earth Simulator, the Atmospheric General Circulation Model with 10 Km mesh could not run on a normal supercomputer. A 30Km to 100Km mesh model was used instead and this could not simulate regional phenomena such as typhoons / hurricanes / cyclones or El Nino / La Nina as these are normally smaller than 100Km. This exemplifies how previous supercomputers didn't have enough power to encapsulate a whole natural phenomenon, so that the simulation had to be done by splitting it into several parts (layers), simulating each part separately and assuming the other parts remained constant.

Initial results are already very exciting. The AGCM code was run using a 10Km mesh simulating the sea surface temperature changes for the whole yearly cycle and the prediction results match very closely the satellite observations. During this experiment the AGCM code achieved 26.58Teraflop/s sustained, i.e. 66.45% of peak performance.

Another experiment concerned humidity and snapshots of precipitation (rainfall) for a year. Again this delivered excellent prediction results showing details of the early stages in the generation of a cyclone. This experiment took 7 days to run to completion on half the system. It run at 14.5Teraflop/s sustained which translates to 72.5% of peak performance of the 320 nodes used. On previous supercomputers, only the outline of the cyclone could be seen and in any case, even if it could be seen it would have taken months to simulate.

Many other examples were presented showing the close prediction of simulated results with real satellite images, including the regression of the ice covering over the Antarctic and Arctic poles.

Seismic wave propagations in Japanese areas were also simulated and compared with real ones. For example, fifty years ago, a major earthquake hit Japan doing a lot of damage especially in the Tokyo area. The wave propagated one way towards Tokyo so the south of Japan was spared. After simulating this event realistically, a new simulation was performed assuming the wave propagates in a radial fashion across the whole of Japan, to assess potential damage this would cause, if it happens in the future. This successful simulation can contribute greatly to disaster relief.

Yet another example is the study of the thermal conductivity of CNT (carbon-nano-tube) and fullerene dynamics. These are extensively studied to find the thermal conductivity for 100 nanometer materials and to search for a super-diamond material of CNT. This simulation allows the design of new devices from new material. It studied material stiffness, which is temperature dependent. In this simulation it was found that CNT is very stable for high temperatures, between 1000 to 4000 degrees.

Other than the above examples, attractive and innovative results are successively produced, e.g., biopolymers and rocket engines. These results confirm that the Earth Simulator project is working well and is in good shape.

Dr Sato added: ?The Earth Simulator possesses the capability for consistently dealing with an entire system that evolves with variety of mutually interacting processes, microscopically and macroscopically. I call this capability ?holistic simulation.? The potential for holistic simulation is superior to experiment and observation. Holistic simulations are a vital means for revealing the true essence of non-equilibrium, nonlinear and open systems, which has been left unexplored by conventional science?.

At the ISC2003 Earth System Modeling session a number of other distinguished speakers presented their work. From the USA: Dr. Warren Washington, Head of the Climate Change Research Section, at NCAR, and Chair of the Science Board, described the Community Climate System Model (CCSM) with an emphasis on IPCC simulations.

He reported on experiments on greenhouse gases, sulfate aerosols, stratospheric ozone, biomass burning and carbon aerosols, land/ vegetation changes, volcanic eruptions, various energy/ emissions use strategies, compared how well the atmospheric model agrees with observations from El Nino and explained plans to include many more features in the models. These include features, such as, improved long wave radiation and clouds, prognostic cloud water formulation, changes to convective precipitation, increased vertical resolution, displacing the North Pole into Greenland, new elastic- viscous- plastic (EVP) for ice, a new thermodynamic model, including multi-category ice thickness scheme, new bio-geophysics formulation, multi-layer soil water and ground T(z) formulation, multi-layer snow model with compaction, river runoff scheme and so on.

Over the next five years, the Climate Change and Assessment Working Group is planning to use CCSM to quantify uncertainty in climate change projections. It identified the need to generate ?forcing repository? so groups can use same ?forcing experiments?. Several steps are to be taken to accomplish this objective: These include, an improved regional climate simulation with a higher resolution atmospheric component, T85, T170 coupled simulations; need more outputs from model in time and space for extremes analyses; probabilistic projections of climate change; ensemble simulations with various forcing constraints and scenarios; integration from 1500-2000 with volcano and solar to look at variability (work with Paleo WG). Understand model response to changes of forcing and climate sensitivity is likely to be the main issue for next IPCC.

There are plans to coordinate experiments involving other models from different modeling centers in addition to CCSM. The plan is to perform a control simulation for 1,000 years, but this will depend on computer time, model data transfer, storage and access in years 2003-05. Dr. Warren Washington concluded that, although he is a Presidential appointee, he does not always agree with U.S. policy, especially opting out of the Kyoto Protocol. As a scientist he wants to contribute in finding an informed answer.

There were also contributions from Europe. Professor Ulrich Cubasch, from the meteorology institute, free university of Berlin, described some of the European activities in climate change studies, supporting the IPCC.

The ?business as usual scenario? (1992) is likely to lead to a climate change with frequent flooding, lack of snow by 2050, no ice at the North pole, melting of glaciers, sea level change, increase in radiation and generally the atmosphere in turmoil. This is why there is a new set of IPCC scenarios formulated and actively under study. These are:

1. A world of rapid economic growth and rapid introduction of new and more efficient technology.

2. A very heterogeneous world with an emphasis on family values and local traditions.

3. A world of ?dematerialization? and introduction of clean technologies.

4. A world with an emphasis on local solutions to economic and environmental sustainability.

These scenarios are so demanding in effort and computing that, each of the G7 developed countries is only able to undertake one or two tasks.

The increased computational demand of these tasks, are estimated to be as follows:

? Increase global resolution to 50km, x 100

? Biology, chemistry, clouds etc, x 100

? Ensembles, x 100

? chaos, multi- member ensembles

? quantifying uncertainty

? Paleo ? simulations (historical), x 1000

? Paleo ? simulations (ice age) x 100,000

? Paleo ? simulations (evolution) x 10** 9

This shows that computing demands for climate change studies go well beyond Petaflop/s.

In Europe there are at least 4 competitive ocean models available, 3 competitive atmosphere models available and so on. This diversity provides strength in terms of science, but not in terms of infrastructure. There are a lot of similarities in these models needing to be unified for efficiency reasons. To stay at the forefront of Earth System Research, Europe needs to come up with a software framework that keeps scientific diversity and at the same time increases process efficiency.

He went on to say, European countries do not always have the same agenda. People want to maintain federal structures, as they are not always willing to relocate to a special center, but prefer to stay where they feel at home. Thus in the European context as a solution for the Earth System Modeling problem the ?One- big- center- provides- everything?, approach does not seem possible.

He concluded: ?Europe is on a promising path in Earth System modeling with regard to models, modelers and national centers of excellence. It plans to maintain scientific diversity and increase efficiency, create a common European Earth System Modeling infrastructure and create a virtual European Earth System Modeling Facility, under the auspices of the PRISM project. What is missing is an ambitious European computing industry?!