The ultrasound research group at the University of North Carolina cherishes the burning ambition to provide the physician with a computer-augmented vision, allowing him to directly see inside the patient's body in a non-invasive manner. At ASCI'98, the annual conference of the Advanced School for Computing and Imaging, which took place at the Vossemeren in Lommel, last June, Dr. Henri Fuchs presented the first preliminary and encouraging results of his team, as well as the challenges the researchers are currently facing to implement augmented reality in real time 3D medical visualization for surgical procedures. The story of a long and winding but fascinating road...
In contrast with virtual reality, which immerses the user in an imaginary world, generated by the computer and cut off from the real world images, augmented reality merges both computer graphics and virtual reality pictures with images of the real world. In an ideal setting, the user gets the impression that real and virtual objects are coexisting. Today, Dr. Fuchs and his team are applying this concept for ultrasound echography imaging. The equipment consists of an optical see-through Head-Mounted Display (HMD) and a Silicon Graphics high performance graphics workstation Onyx with RealityEngine2 (RE2), supplied in addition with a Sirius real time video capture unit. It offers the doctor Superman's X-ray vision ability to look into the body without using harmful radiation.
The ultrasound group used the system to scan a foetus inside the mother's womb by means of an ultrasonic sensor. Next, a three-dimensional model of the foetus was introduced on top of the womb. Multiple ultrasound slices of the foetus were visualized through volume rendering for real time display in the virtual environment. The research team has been using four different systems for ultrasound visualization up till now. The first three systems were limited to passive obstetrics examinations but the recent Onyx RE2 configuration should be powerful enough to serve in diagnostic procedures, such as ultrasound-guided needle biopsies, and in cardiology.
The system's frame grabber is able to capture both HMD camera and ultrasound video. While the camera video is showed in the background, the ultrasound images are transmitted into texture memory and displayed on polygons, that are emitted by the ultrasound probe inserted into a synthetic opening of the scanned patient's body. The physician has to specifically train his hand-eye coordination and his skill for 3D viewing, as to correctly perform a needle biopsy. In the new Onyx RE2 system, the ultrasound slices are not reconstructed into a volume anymore, but directly displayed instead as transparent polygons, carrying the mappings of the ultrasound video images.
The image quality looks very promising but still isn't perfect yet. The Fuchs team therefore aims to acquire a superior ultrasound machine, which allows to create image slices thinner than the current 1-2 cm thickness, in order to avoid a blurry vision. Real time interactive rendering of individual slices in the virtual environment constitutes a second problem. The researcher group envisages to build a new system, which will reconcile the excellent real time performance of the first system with the image texture rendering capacities of the Onyx RE2. A third challenge lies in the optimization of the calibration techniques to properly align the real and virtual objects in relation to each other, to create the illusion of perfect coexistence without any perceived lag or spatial misregistration.
Dr. Fuchs insists that augmented reality systems, combined with ultrasound techniques and medical visualization, are not quite ready yet for daily use in clinical environments. Surgical procedures indeed imply the highest level of precision. Consequently, the ultrasound group has made a first priority of the search for alternative tracking systems, to allow a seamless merging of the virtual universe within the real medical world. To get a feel of what you can see with Superman's "X-ray eyes", you can explore the illustrated world of computer-augmented vision technology at the University of North Carolina Web site.