1st Workshop

Distributed Virtual Supercomputers

Hotel de Santa Fe — Santa Fe, New Mexico

June 24 – 27, 1997

The concept of Distributed Supercomputing (DS) is to interconnect commodity computing building blocks to create multi-Teraflop capability/capacity engines. Sandia is exploring DS as a complement to large-scale vendor machines because we see the development of a "new computing industry" — companies like Intel are moving toward horizontal integration and away from chip-to-supercomputer vertical integration.

Connecting commodity components, potentially from multiple vendors, with commodity interconnects and software poses many challenges and provides many opportunities. The intent of this joint Sandia/Los Alamos/Oakridge/Swiss workshop will be to explore these issues such as:

  • Is there a cost advantage to distributed supercomputing?
  • What are the implications for software at every level?
  • Does DS change the needs at the desktop?
  • Can we leverage the academic work on systems like Beowulf?
  • Can we turn the abundance of computational capacity we will have in the coming decade into a similar abundance of computational capability?
  • What applications can easily use such a system?
  • What changes would have to be made to other applications to allow them to use such a system?

Our intent is to keep the workshop informal with lots of room for free flowing discussion.


June 25th: Issues, Questions, Challenges

8:15 – 9:00Breakfast and round the room introductions
9:00 – 9:30Welcome and meeting purpose.
     William Camp, SNL
9:30 – 11:30Hardware issues in Distributed Supercomputing
     James Tomkins, SNL
     Karl-Heinz, LANL
     Anton Gunzinger, ETH
     Al Geist, ORNL
11:30 – 1:00Lunch
1:00 – 3:00Software issues in Distributed Supercomputing
     Richard Barrett, LANL
     Pierre Kuonen, ETH
     Tim Sheehan, ORNL
     David Greenberg, SNL
3:00 – 3:15Break
3:15 – 5:15Applications issues in Distributed Supercomputing
     Ralf Gruber, EPFL
     Ken Kliewer, ORNL
     TBD, LANL
     Robert Clay, SNL

June 26th: Current Efforts

8:00 – 8:30Breakfast
8:30 – 10:30The computational plant project at SNL
     Art Hale, Overview
     Robert Clay, A potential hardware system design
     Rolf Riesen, A potential system software architecture
     Pang Chen, Providing balanced, usable I/O
     Lee Ann Fisk, Supplying a runtime environment
     David Evensky, Scalable Object Systems for Management and Control
10:30 – 10:45Break
10:45 – 11:45Session A, (Kiva A Room)
     Al Geist, ORNL, Frontiers of PVM
Session B, (Kiva B Room)
     Simulating Fluid Dynamics and Turbulence I
     Michel Deville, EPFL "Parallel Spectral Elements for Complex Fluids"
     Mark Christon, SNL
     John Shadid, SNL
11:45 – 1:00Lunch, (Hotel Patio)
1:00 – 2:30Session A, (Kiva A Room)
     Michael Warren, LANL,
          "Experiences with Loki"
     Ian Philp, LANL
          "Obstacles to providing low-latency,
          high-bandwidth networking in
          pheterogeneous and wide-area environments"
Session B, (Kiva B Room)
     Simulating Fluid Dynamics and Turbulence I
     Open discussion
1:00 – 2:30Session C, (Suite 225)
     Electronic Structure Calculations of Materials
     Ellen Stechel, SNL
     Grant Heffelfinger, SNL
     Alan Wright,SNL
     Heinrich Roder, LANL
     Roberto Car, EPFL
     Dwight Jennison, SNL
     Jonathan Boettger, LANL
     Mark Sears, SNL
     Peter Schultz, SNL
     Olof Eriksson, LANL
      Jeff Nelson, SNL
     Roland Stumpf, SNL
     Rick Martin, LANL
2:30 – 3:00Break (Kiva A Room)
3:00 – 4:30Session A, continued, (Kiva A Room)
     Anton Gunzinger
         Network for Parallel Commodity Computing
     Ralf Gruber
         The Scientific Macintosh
Session B, continued, (Kiva B Room)
     Adaptive Mesh Refinement
     Roland Richter, EPFL
         Unstructured Mesh Adaption Within a
         True Parallel Environment
     Mike Wong, SNL
     Karen Devine, SNL
     Joe Jung, SNL
     Walt Witkowski, SNL
     James Peery, SNL
Session C, continued (Suite 255)

June 27: Preparing for the Future, (Kiva A Room)

8:30-12:00Open discussion of what the key design decisions are that will need to be made, plans to address key bottlenecks, and plans to explore alternatives. Jim Tomkins, Art Hale, Karl-Heinz Winkler, Anton Gunzinger, Ken Kleiwer, Rob Armstrong will lead discussion.
12:00-1:00Lunch and wrap up (Kiva B Room)


Swiss Federal Institute of Technology

Roberto Car
Michel Deville
Rolf Gruber
Anton Gunzinger
Pierre Kuonen
Silvio Merazzi
Roland Richter
Marie-Christine Sawley

Sandia National Laboratories

Rob Armstrong
R. Michael Cahoon
William Camp
Robert Clay
David Evensky
David Greenberg
Grant Heffelfinger
Bruce Hendrickson
Rob Leland
John Noe
Tom Pratt
Mark Sears
Jim Tomkins
Michael Vahle
Lucille Verdugo
Pete Wycoff

Oak Ridge National Laboratory

Al Geist
Ken Kliewer
Tim Sheehan


The Distributed Virtual Supercomputing workshop was held from June 25 – 27, 1997, in Santa Fe, New Mexico. The workshop was coordinated by Sandia in response to their new ASCI project devoted to distributed computing.

People from Sandia, ORNL, LANL, and a couple of Swiss groups participated. Pretty cool deal: build a big time computational machine using commodity parts. Plan for expansion. Plan for widely distributed computing platforms. Below is a partial description of what was discussed.

Some of the participants:

    Sandia: Bill Camp, Jim Tomkins, David Greenberg
    ORNL: Al Geist, Ken Kliewer, Tim Sheehan
    Swiss: Anton Gutzinger (ETH) and Ralf Gruber (EPFL)
    LANL: Richard Thomsen, Ian Philp, Mike Warren (Omissions are my oversight. Sorry.)

Sandia and ORNL described the type of applications they are interested in running on such a machine.

Sandia folks described their "Computational Plant" project. The ’97 goal is to build a 100 GFlops peak/60 GFlops achieved machine. Here are some details:

Hardware (presented by Robert Clay)


  • DEC chip (433 or 500MHz)
  • Myrinet networking boards. Will explore topologies, but a mesh seems the likely candidate in order to allow for easier growth.
  • Flexible partition a la ASCI Red: service, system, compute, I/O.
  • 1 I/O node per scalable unit (16 cpus?)

System software (presented by Rolf Riesen)

  • Lightweight operating system kernel (initially linux, then puma/cougar port to alpha)
  • Portals for fast communication
  • Big compiler effort.
  • MPI (details below)

Other stuff

Heterogeneity (Al Geist)

Al showed that we have most of the pieces to allow for heterogeneity. They just need to be adapted/adopted/refined. Many presenters/participants shared this view. However, a homogeneous setup will be pursued in the initial phases.

Long distance message passing (Tim Sheehan)

A collaboration between ORNL, Sandia, and the Pittsburg Supercomputing center involves running applications across widely distributed computing resources. With a lot of work, codes (CTH, LSMS) were executed in parallel on machines located at Sandia and ORNL. For details, seehttp://www.lanl.gov/organization/cic/cic8/para-dist-team/DVS/paragon_results.html.

Many argue that this type of computing is simply a political exercise. Regardless, the challenges involved in computing across the country are related to the challenges involving computing between two machines located in the same room. Bill Camp: "The alternative to having two major computing resources located at two sites is to locate them both at one site. Which would you prefer?" Answer: "If they are located at my site, I’m all for the single site model." (My argument for that site being LANL is that the speed of light is faster at altitude.)

Message passing

Although there were some rumblings regarding other programming models, it appears that explicit message passing is the clear front runner. And probably will be an MPI implementation, although only a subset of MPI’s functionality will get attention with respect to optimization. (That is, adapt a public domain version of MPI, such as mpich or lam.) Hardware folks are quite interested in providing an excellent network infrastructure. In fact, there is talk of not only implementing barriers in hardware, but also reductions/broadcasts/gather/scatter. This could be a big win.

A variety of subgroups will be formed in order to attack the multitude of challenging issues that must be addressed/conquered in order for this project to succeed. It could be argued that these issues are not applicable to us rich folk who can afford $150,000,000 machines, but I would counter that it is exactly these types of issues that must be addressed in order to kick us up to the 10/30/100 TFLOP machines, regardless of cost.

Details regarding how to get in on such discussions should appear shortly. Should be fun…

Richard Barrett

What is it?

"It’s like a tree. It grows, and is pruned to achieve scalability."
    – Jim Tomkins

"You start with a tree. Then you get a forest. Then you have to deal with a continent."
    – Al Geist

"It’s like a cow: we feed it, we milk it, then we slaughter it."
    – I ain’t sayin’…