Sandia’s ‘being there’ visual hardware enhances long-distance collaborations

If a surgeon in New York wants the opinion quickly of a specialist in Cairo, she probably would send medical X-ray or MRI files as e-mail attachments or make them accessible in Internet drop zones.

But jointly viewing and interacting with the images — a more effective way to discuss problems — currently takes minutes for each turn of a visualization. This could be too time-consuming to help a patient on the operating table. In less extreme cases, with medical specialists being paid by the clock, the time delays during extensive consultations could soon lead, as the late Senator Everett Dirksen put it, to real money.

Now a team of Sandia engineers has applied for a patent on interactive remote visualization hardware that will allow doctors (or engineers, or oil exploration teams, or anyone else with a need to interact with computer-generated images from remote locations) to view and manipulate images as though standing in the same room. The lag time between action and visible result is under 0.1 second even though the remote computer is thousands of miles away.

"The niche for this product is when the data set you’re trying to visualize is so large you can’t move it, and yet you want to be collaborative, to share it without sending copies to separate locations," says Sandia team leader Lyndon Pierson (9336). "We expect our method will interest oil companies, universities, the military — anywhere people have huge quantities of visualization data to transmit and be jointly studied."

He adds, "Significant commercial interest [in the new device] has been demonstrated by multiple companies."

The Sandia hardware leverages without shame the advances in 3-D commercial rendering technology "in order not to re-invent the wheel," says Perry Robertson (1751).

Graphics cards for video games have extraordinary 2-D and even 3-D rendering capabilities but exercise them only inside the cards. These images are then fed to the nearby monitor — a cozy arrangement that does not solve the problem of how to plug visuals formatted for 60 images a second into a network, says Perry.

Fortunately, the Sandia extension hardware looks electronically just like a monitor to the graphics card, says Perry. "So, to move an image across the Internet, as a first step our device grabs the image."

The patented Sandia hardware squeezes the video data flooding in at nearly 2.5 gigabits a second into a network pipe that carries less than 0.5 gigabits/sec.

"While compression is not hard, it’s hard to do fast. And it has to be interactive, which streaming video typically is not," says Lyndon.

The Sandia compression minimizes data loss to ensure image fidelity. "Users need to be sure that the things they see on the screen are real, and not some artifact of image compression," says Lyndon.

The group knew that a hardware solution was necessary to keep up with the incoming video stream. "Without it, the receiver’s frame rate would be unacceptably slow," says Perry. "We wanted the user to experience sitting right at the supercomputer from thousands of miles away."

"In an attempt to reduce the need for additional hardware," says John Eldridge (9336), who wrote the software applications, "we also created software versions of the encoder and decoder units for testing purposes. However, there is only so much you can do in software at these high resolutions and frame rates."

The custom-built apparatus has two boards — one for compression, the other for expansion. The boards use standard low-cost SDRAM memory, like that found in most PCs, for video buffers. Four reprogrammable logic chips do the main body of work. A single-board PC running Linux is used for supervisory operations. "We turned to Linux because of its networking support and ease of use," says Ron Olsberg (9336), project engineer.

"We built this apparatus for very complex ASCI visualizations. If we could have bought it off the shelf, we would have," says Perry.

Funded by ASCI’s [Advanced Scientific Computing Initiative] Problem-Solving Environment, a pair of boards cost about $25,000, but are expected to cost much less when commercially available.

A successful demonstration took place in late October between Chicago and the Amsterdam Technology Center in the Netherlands. A second demonstration occurred between Sandia locations in Albuquerque and Livermore and the show floor of the Supercomputing 2002 convention in Baltimore in November.

"Now that this technology is out there, we expect other applications will begin to take advantage of it," says Lyndon. "Their experiences and improvements will eventually feed back into US military capability."

In addition to Perry, John, Lyndon, and Ron, the design team also included Karl Gass (1751) and Tom Pratt and Edward Witzke (both 9336).