Distributed Virtual Supercomputers
Hotel de Santa Fe — Santa Fe, New Mexico
June 24 – 27, 1997
The concept of Distributed Supercomputing (DS) is to interconnect commodity computing building blocks to create multi-Teraflop capability/capacity engines. Sandia is exploring DS as a complement to large-scale vendor machines because we see the development of a "new computing industry" — companies like Intel are moving toward horizontal integration and away from chip-to-supercomputer vertical integration.
Connecting commodity components, potentially from multiple vendors, with commodity interconnects and software poses many challenges and provides many opportunities. The intent of this joint Sandia/Los Alamos/Oakridge/Swiss workshop will be to explore these issues such as:
- Is there a cost advantage to distributed supercomputing?
- What are the implications for software at every level?
- Does DS change the needs at the desktop?
- Can we leverage the academic work on systems like Beowulf?
- Can we turn the abundance of computational capacity we will have in the coming decade into a similar abundance of computational capability?
- What applications can easily use such a system?
- What changes would have to be made to other applications to allow them to use such a system?
Our intent is to keep the workshop informal with lots of room for free flowing discussion.
June 25th: Issues, Questions, Challenges
|8:15 – 9:00||Breakfast and round the room introductions|
|9:00 – 9:30||Welcome and meeting purpose.|
William Camp, SNL
|9:30 – 11:30||Hardware issues in Distributed Supercomputing|
James Tomkins, SNL
Anton Gunzinger, ETH
Al Geist, ORNL
|11:30 – 1:00||Lunch|
|1:00 – 3:00||Software issues in Distributed Supercomputing|
Richard Barrett, LANL
Pierre Kuonen, ETH
Tim Sheehan, ORNL
David Greenberg, SNL
|3:00 – 3:15||Break|
|3:15 – 5:15||Applications issues in Distributed Supercomputing|
Ralf Gruber, EPFL
Ken Kliewer, ORNL
Robert Clay, SNL
June 26th: Current Efforts
|8:00 – 8:30||Breakfast|
|8:30 – 10:30||The computational plant project at SNL|
Art Hale, Overview
Robert Clay, A potential hardware system design
Rolf Riesen, A potential system software architecture
Pang Chen, Providing balanced, usable I/O
Lee Ann Fisk, Supplying a runtime environment
David Evensky, Scalable Object Systems for Management and Control
|10:30 – 10:45||Break|
|10:45 – 11:45||Session A, (Kiva A Room)|
Al Geist, ORNL, Frontiers of PVM
|Session B, (Kiva B Room)|
Simulating Fluid Dynamics and Turbulence I
Michel Deville, EPFL "Parallel Spectral Elements for Complex Fluids"
Mark Christon, SNL
John Shadid, SNL
|11:45 – 1:00||Lunch, (Hotel Patio)|
|1:00 – 2:30||Session A, (Kiva A Room)|
Michael Warren, LANL,
"Experiences with Loki"
Ian Philp, LANL
"Obstacles to providing low-latency,
high-bandwidth networking in
pheterogeneous and wide-area environments"
|Session B, (Kiva B Room)|
Simulating Fluid Dynamics and Turbulence I
|1:00 – 2:30||Session C, (Suite 225)|
Electronic Structure Calculations of Materials
Ellen Stechel, SNL
Grant Heffelfinger, SNL
Heinrich Roder, LANL
Roberto Car, EPFL
Dwight Jennison, SNL
Jonathan Boettger, LANL
Mark Sears, SNL
Peter Schultz, SNL
Olof Eriksson, LANL
Jeff Nelson, SNL
Roland Stumpf, SNL
Rick Martin, LANL
|2:30 – 3:00||Break (Kiva A Room)|
|3:00 – 4:30||Session A, continued,
(Kiva A Room)|
Network for Parallel Commodity Computing
The Scientific Macintosh
|Session B, continued, (Kiva B Room)|
Adaptive Mesh Refinement
Roland Richter, EPFL
Unstructured Mesh Adaption Within a
True Parallel Environment
Mike Wong, SNL
Karen Devine, SNL
Joe Jung, SNL
Walt Witkowski, SNL
James Peery, SNL
|Session C, continued (Suite 255)|
June 27: Preparing for the Future, (Kiva A Room)
|8:30-12:00||Open discussion of what the key design decisions are that will need to be made, plans to address key bottlenecks, and plans to explore alternatives. Jim Tomkins, Art Hale, Karl-Heinz Winkler, Anton Gunzinger, Ken Kleiwer, Rob Armstrong will lead discussion.|
|12:00-1:00||Lunch and wrap up (Kiva B Room)|
Swiss Federal Institute of Technology
Sandia National Laboratories
R. Michael Cahoon
Oak Ridge National Laboratory
The Distributed Virtual Supercomputing workshop was held from June 25 – 27, 1997, in Santa Fe, New Mexico. The workshop was coordinated by Sandia in response to their new ASCI project devoted to distributed computing.
People from Sandia, ORNL, LANL, and a couple of Swiss groups participated. Pretty cool deal: build a big time computational machine using commodity parts. Plan for expansion. Plan for widely distributed computing platforms. Below is a partial description of what was discussed.
Some of the participants:
Sandia: Bill Camp, Jim Tomkins, David Greenberg
ORNL: Al Geist, Ken Kliewer, Tim Sheehan
Swiss: Anton Gutzinger (ETH) and Ralf Gruber (EPFL)
LANL: Richard Thomsen, Ian Philp, Mike Warren (Omissions are my oversight. Sorry.)
Sandia and ORNL described the type of applications they are interested in running on such a machine.
Sandia folks described their "Computational Plant" project. The ’97 goal is to build a 100 GFlops peak/60 GFlops achieved machine. Here are some details:
Hardware (presented by Robert Clay)
- DEC chip (433 or 500MHz)
- Myrinet networking boards. Will explore topologies, but a mesh seems the likely candidate in order to allow for easier growth.
- Flexible partition a la ASCI Red: service, system, compute, I/O.
- 1 I/O node per scalable unit (16 cpus?)
System software (presented by Rolf Riesen)
- Lightweight operating system kernel (initially linux, then puma/cougar port to alpha)
- Portals for fast communication
- Big compiler effort.
- MPI (details below)
Heterogeneity (Al Geist)
Al showed that we have most of the pieces to allow for heterogeneity. They just need to be adapted/adopted/refined. Many presenters/participants shared this view. However, a homogeneous setup will be pursued in the initial phases.
Long distance message passing (Tim Sheehan)
A collaboration between ORNL, Sandia, and the Pittsburg Supercomputing center involves running applications across widely distributed computing resources. With a lot of work, codes (CTH, LSMS) were executed in parallel on machines located at Sandia and ORNL. For details, seehttp://www.lanl.gov/organization/cic/cic8/para-dist-team/DVS/paragon_results.html.
Many argue that this type of computing is simply a political exercise. Regardless, the challenges involved in computing across the country are related to the challenges involving computing between two machines located in the same room. Bill Camp: "The alternative to having two major computing resources located at two sites is to locate them both at one site. Which would you prefer?" Answer: "If they are located at my site, I’m all for the single site model." (My argument for that site being LANL is that the speed of light is faster at altitude.)
Although there were some rumblings regarding other programming models, it appears that explicit message passing is the clear front runner. And probably will be an MPI implementation, although only a subset of MPI’s functionality will get attention with respect to optimization. (That is, adapt a public domain version of MPI, such as mpich or lam.) Hardware folks are quite interested in providing an excellent network infrastructure. In fact, there is talk of not only implementing barriers in hardware, but also reductions/broadcasts/gather/scatter. This could be a big win.
A variety of subgroups will be formed in order to attack the multitude of challenging issues that must be addressed/conquered in order for this project to succeed. It could be argued that these issues are not applicable to us rich folk who can afford $150,000,000 machines, but I would counter that it is exactly these types of issues that must be addressed in order to kick us up to the 10/30/100 TFLOP machines, regardless of cost.
Details regarding how to get in on such discussions should appear shortly. Should be fun…
What is it?
"It’s like a tree. It grows, and is pruned to achieve scalability."
– Jim Tomkins
"You start with a tree. Then you get a forest. Then you have to deal with a continent."
– Al Geist
"It’s like a cow: we feed it, we milk it, then we slaughter it."
– I ain’t sayin’…