Getting into colliding black holes and neutron stars

Mannheim, 21-6-97 Eighty years ago, Einstein developed the theory of general relativity. Now the largest parallel supercomputers are approaching the speed and memory requirements to solve the complete set of Einstein's equations. Full 3D simulation of events like colliding black holes and neutron stars are within reach. Ed Seidel, Max-Planck Institute for gravitational physics at Potsdam, Germany reported at the Mannheim Supercomputer Seminar. Parallelization strategies and performance issues on various machines, including SGI/Cray Origin 2000, Cray T3D/E and C-90, Convex Exemplar, IBM SP-2, and Thinking Machines CM-5 show the progress made thusfar.

Einstein's equations for the structure of spacetime are a set of fourteen highly complex, coupled, nonlinear partial differential equations. Most of what is known about this fundamental theory of Physics came from linearized solutions, highly idealized solutions possessing a high degree of symmetry (e.g., static, or spherically or axially symmetric), or from perturbations of these solutions. Over the last 30 years a growing research area, called Numerical Relativity, has developed, where computers produce numerical solutions to these equations. Although much has been learned through this approach, progress was slow due to the complexity of the equations and inadequate computer power. The 3D spiraling coalescence of two black holes or neutron stars needs for an accurate 3D simulation in the order of 100,000 Cray Y-MP hours.

Seidel hopes that this problem can be solved within the next decade. Scalable parallel computers are quickly revolutionizing computational science, and numerical relativity is a great beneficiary of these developments. They developed several 3D codes to solve the complete set of Einstein equations, some of them ran between 10 and 15 GFlops on machines like the 512 node Thinking Machines CM-5 or Cray T3D, about 200 times faster than 2D codes on the Cray Y-MP (which has only about 3% the memory capacity of the CM-5). A new version of this code promises about five times this performance on similarly sized Cray T3E and Origin 2000 machines available this year. Such machines are expected to scale up rapidly as faster processors are connected together in even higher numbers, achieving Teraflop performance in a few years. When such machines are available they hope to be able to tackle problems like the 3D Einstein equations for the gravitational field routinely.

Black holes

Black holes are thought to be formed from the collapse of very massive stars after they have exhausted their nuclear fuel. If the remnant star is massive enough, its self-gravity will overwhelm any pressure support and it will collapse to a singularity where its density and other physical quantities become infinitely large. Just outside this singularity, gravity is strong enough to trap all signals, including "outgoing'' light rays. Further away from the singularity, where gravity is weaker, light rays can escape. There is a critical boundary, called the event horizon, where outgoing light rays never escape, nor fall into the black hole. As many stars exist as binary pairs, it is expected that in some cases both stars will collapse to form black holes with singularities hidden inside horizons.

The study of black hole binary systems, and detecting and interpreting signals that they emit, is the focus of major research initiatives of different groups worldwide. Some develop high performance codes for general relativistic hydrodynamics to study the problem of coalescing neutron stars. Fully relativistic hydrodynamics is just beginning.

First Seidel discussed axisymmetric simulations of a colliding black hole system, and then showed simulations of the same system performed with a fully 3D code, written in cartesian coordinates. The axisymmetric simulations demonstrated that the two holes, initially at rest, fall to each other, collide and merge into a highly distorted single black hole. This newly formed black hole then vibrates, emitting gravitational waves in the process.

With these simulations additionally the wave forms from this highly nonlinear process can be described rather accurately by perturbation theory. One might expect the system to undergo nonlinear oscillations in such a violent event, but the final wave forms are actually fit very accurately by the quasinormal modes of the final black hole, known from linear perturbation theory. Furthermore, in the regime where the black holes were separated by a large distance, the results agree well with an approximation that considers one black hole to be a test particle falling into the other black hole!

Seidel presented scientific and timing results on different machines based on the "old" SIMD-based code H3expresso. On Thinking Machines CM5 (512 processors) they reached 12 GFlop/s, in the first step 7 GFlop/s on a Cray T3E (512 processors). With Cray support they improved the performance to 10.4 Gflop/s now. On the C90 (16 processors) they reached 7.5 Gflop/s, on an IBM SP2 (256 processors) 10 Gflop/s. On cache-based machines the results drop down to 0.3 Gflop/s on an Hewlett-Packard Exemplar (8 processors) and to 0.9 Gflop/s on an SGI PowerChallenge (16 processors).

From scratch Seidel developed Cactus - using the experiences of parallelism and single processor performance. It is not written in a data-parallel fashion, but obtains parallelism through MPI and is targetted to cache-based systems. With their Cactus program, Seidel tried to deal with the difficult issues of code maintenance and management.

Collaborators at 7 institutions are using the code for various research projects. To allow the code to stay functional with a large group of developers, they have adopted a "plug-in" style coding technique. The Cactus code has, at its core, the "Flesh" of the cactus. This contains the basic Einstein solver, the parallel domain decomposition software, I/O, and a few utilities. The "Flesh" is about 6000 lines of highly optimised C and Fortran, not counting utility libraries, makefile, and Perl scripts.

More information and results can be obtainedat theMax Planck Insitute's web site.

Uwe Harms