Advanced Simulation and Computing

Advanced Systems Technology Test Beds

Advanced Simulation and Computing (ASC) project, Compton, Curie, Shannon, Teller

As part of NNSA’s Advanced Simulation and Computing (ASC) project, Sandia has acquired a set of advanced architecture test beds to help prepare applications and system software for the disruptive computer architecture changes that have begun to emerge and will continue to appear as HPC systems approach Exascale.  In contrast to ASC Advanced Technology or Commodity Technology supercomputer platforms, these test bed systems are not for production computing cycles.  Instead, they are intended to be pre-production or first-of-a-kind prototypes to support exploration of a diverse set of architectural alternatives that are possible candidates for future pre-Exascale systems.  While these test beds can be used for node-level exploration they also provide the ability to study inter-node characteristics to understand future scalability challenges.  To date, the test bed systems populate 1-6 racks and have on the order of 50-200 multi-core nodes, many with an attached co-processor or GP-GPU.

The test beds allow for path finding explorations of 1) alternative programming models, 2) architecture-aware algorithms, 3) energy efficient runtime and system software, 4) advanced memory sub-system development and 5) application performance. But that is not all.  Validation of computer architectural simulation studies can also be performed on these early examples of future Exascale platform architectures.  As proxy applications are developed and re-implemented in architecture-centric versions, the developers need these advanced architecture systems to explore how to adapt to an “MPI + X” paradigm, where “X” may be more than one disparate alternative.  This in turn, demands that tools be developed to inform the performance analyses.  ASC has embraced a co-design approach for its future advanced technology systems.  By purchasing from and working closely with the vendors on pre-production test beds, both ASC and the vendors are afforded early guidance and feedback on their paths forward.  This applies not only to hardware, but other enabling technologies such as system software, compilers, and tools.

There are currently several test beds available for use, with more in planning and integration phases. They represent distinct architectural directions and/or unique features important for future study. Examples of the latter are custom power monitors and on-node solid state disks (SSD).

Feature
Intel Xeon Phi
Cray XK7
NVIDIA
AMD Fusion
Cray XC30m
Host name
Compton
Curie
Shannon
Teller
Volta
CPU

Dual Socket Intel Xeon E5-2670
(Sandy Bridge)
2.6 GHz
8-core

AMD Opteron Interlagos
2.1 GHz
16-core

Dual Socket Intel Xeon E5-2670 (Sandy Bridge)
8-core

AMD A10-5800K (Piledriver)
3.8GHz
Quad-core

Intel Xeon E5-2695 V2 dual socket (Ivy Bridge)
total 24 cores
2.4 GHz

Accelerator

Pre-Production Intel Xeon Phi Co-processor
(codenamed Knights
Corner)
2 per node

Nvidia
Kepler K20X

Nvidia
Kepler K20X
2 per node

Radeon HD-7660D (Northern Islands) with on-die integration

None
Cores per Accelerator / Co-Processor

57
1.1GHz cores

2688
732 MHz cores

2688
732 MHz cores

384
800MHz cores

N/A
Nodes

42

52

32

104

56
Interconnect

Mellanox
Quad Data Rate Infiniband

Gemini

Mellanox
Quad Data Rate Infiniband

QLogic
Quad Data Rate Infiniband

Aries
Other

80GB SSD
per node

Full featured monitoring and control system and restricted access

Full PCI
Generation 3
NVIDIA GPU Direct

Integrated CPU/GPU+ 256GB SSD and a
custom-designed power monitoring capability

Full featured RAS system including power monitoring and control capabilities

Test Bed Team Members

The test bed team members from left to right. Back row: Sue Kelly, Jim Brandt, Jim Laros, and Victor Kuhns. Front row: Char Arias, Bob Balance, and Jason Repik. Ann Gentile was not present. The team is standing in Sandia’s computer hardware museum.
The test bed team members from left to right. Back row: Sue Kelly, Jim Brandt, Jim Laros, and Victor Kuhns. Front row: Char Arias, Bob Ballance, and Jason Repik. Ann Gentile was not present. The team is standing in Sandia’s computer hardware museum.

The WIKI site is restricted to account holders with a Sandia crypto card.