At the Electronics Laboratory of the Swiss Federal Institute of Technology (ETH) in Zürich, research assistant Alex Rhomberg is involved in the laparoscopy project, called LASSO. During the Parco 99 Conference, he provided an overview of the parallel architecture which has been developed in the project to perform interactive finite element method (FEM) computations for the simulations of organ deformations in a surgery simulator. Laparascopy is an operation technique for minimally invasive surgery. The basic idea is to minimize damage to healthy tissue while trying to reach the actual surgical location. This results in major gain in patient recovery after operation.
Laparoscopy technique is difficult to learn for the surgeon since he only disposes of a monocular view through a wide-angle lens in front of the camera and he hasn't but instruments with long handles which are not easy to handle. Currently, trainee surgeons are learning on boxes with foam rubber organs in which they manipulate their instruments. Moreover, pigs are used to operate on, which is not very good from an ethical point of view. Therefore, the Swiss group is aiming at developing a simulator with virtual reality technology. The surgeon holds the force feedback handles while their position is transmitted to the finite element model in order to calculate the response of the human organs' movement. The surgeon receives a realistic pressure feedback as he would in an actual operation.
Since the surgeon is working in real time, the system has to function in real time also. The LASSO project includes several activities, consisting of the measurement of finite element parameters and the generation of a model. At present, there is no accurate model available of the lower abdomen while the material properties of the organs are unknown. The computational problem is non-linear whereas the materials are not homogeneous. There equally are large deformations and displacements. Another task involves the generation of photorealistic graphics in real time from a wet surface. The force feedback device needs several degrees of freedom. More than one of these instruments have to be put closely together, allowing the surgeon to use both hands in the operation. The team of Alex Rhomberg in particular is working on the finite element calculations in real time.
The finite element model consists of elements and nodes. The elements are deformed and exercise extra force on the nodes. In every timestep, the force that acts on the node to push it into a new position is calculated. The nodes are gathered to calculate the inner forces and scattered back to their initial position afterwards. These gather-and-scatter operations are important as soon as elements are distributed on different computers. The elements are distributed in subdomains that in turn are assigned to the processors in the parallel computing architecture. Some of the nodes in the mesh might be on different processors, so the forces or the positions have to be communicated if any changes occur. These communications are locally constrained along the division borders.
In order to decrease the number of communications, simple calculations are duplicated. The determination of the inner forces of the element forms the computationally most intensive part so this is only performed once for every element. Each processor calculates the sum of forces for the node which it represents. The system is highly scalable which means that a processor can be used for each element but it is also closely coupled because within every time step, all the communications have to be finished. One single time step lasts 100 microseconds. Therefore, the LASSO team has to decrease latency. The calculation requirement is about 15 Mflops per second for every element. For every node, 240 Kbytes of data per second have to be exchanged with the processor on the other side of the communication link, for which a rate of 30 Mbytes per second per processor is needed.
The system topology matches the 3D finite element model since the elements are distributed to a 3D lattice of processors. High performance workstations are used as processing nodes whereas the surgery simulator is developed on an SGI Onyx2 8-processor computer, that allows concurrent software design. It is important for the parallel calculation to hide the communication latency so computation and communication run concurrently until the end of each time step. To this purpose, the LASSO team has introduced both static load balancing and scheduling in the system, which consists of 16 alpha 21264 processors connected to a Myrinet network and Fast Ethernet. The Myrinet processor is able to generate and receive data packets by using direct virtual channels.
In the local communication, there is no overhead for the Central Processing Units (CPUs) which just store and load data in the PCI memory for the NIC to send it off. The Myrinet sends the packet through the switched network and the receiving card extracts the data to store it at the appropriate address in the user space. As long as communication is faster than computation, data exchange is hardly more expensive than local memory accesses. The LASSO prototype achieves a latency of 25 microseconds and a bandwidth of 45 MB per second for a packet size of 512 bytes. Double buffering enables the use of loose coupling while still guaranteeing the integrity of the data sent. Since there is symmetric communication, the team can use all the communication for the synchronisation between processors.
Real time finite element is needed for accurate physical representation of the human organs in the body. The method poses high requirements for parallel computing but network congestion is reduced by exploiting the fact that data communication is locally constrained. The strength of the LASSO system is lying in the communication interface which can perform at a high bandwidth rate while showing a low latency. For those who want to know more about the parallel architecture for interactive finite element model computations, we gladly refer to the home page of the LASSO project.