Sandia’s parallel DSMC codes yield unprecedented physics, engineering insights

MIR PROJECTION — Steve Plimpton (1444), left, and Michael Gallis (1513) look at a projection of a model of the Russian MIR space station, which fell out of orbit several years ago and disintegrated, with the remains ending up at the bottom of the Indian Ocean. Using Sandia’s 3-D code SPARTA, the calculation is simulating an instance of the process of de-orbiting. (Photo by Randy Montoya)

by Sue Major Holmes

When the space shuttle Columbia disintegrated in the first stage of re-entry in 2002, sophisticated computer models were key to determining what happened.

A piece of foam flew off at launch and hit a tile, damaging the leading edge of the shuttle wing and exposing the underlying structure. Temperatures reached thousands of degrees as Columbia plunged toward Earth at 27 times the speed of sound, says Sandia fluid science and engineering researcher Michael Gallis, who used NASA codes and a Sandia DSMC code called Icarus to do accident simulations that proved critical to investigators.

But researchers realized a more sophisticated code would be even more valuable. Now Michael (1513) and computational scientist Steve Plimpton (1444) have created a parallel three-dimensional DSMC code called SPARTA. In July, Sandia released it as open source, available at In addition, Michael presented results of work using SPARTA at an invited keynote lecture at the 29th annual International Conference on Rarefied Gas Dynamics.

Three-D codes like SPARTA represent physical reality more accurately than 2-D codes such as Icarus. More accurate simulations mean designers can account for many more details in new spacecraft or satellites. However, there’s a price for greater physical realism. “A 3-D simulation is like a series of 2-D ones, sometimes making it thousands of times more demanding,” Michael says.

DSMC codes simulate molecules moving and bouncing off each other and objects, just as they do in real gas flows. Underlying statistics determine when and how molecular collisions occur, enabling predictions of energy transfer and chemical reactions. The DSMC approach typically is used to model low-pressure gases. Physical problems where gas is at low pressure are less common than problems with gas at higher pressures.

“Monte Carlo” refers to the randomized way in which collision parameters are chosen for pairs of particles, based on statistical principles. The order in which molecules collide is random, but not the rate or outcome of a large number of collisions, which can be described by well-known mathematical models.

Sandia recognized DSMC potential early

More than 20 years ago, Tim Bartel (6233) and Steve Plimpton developed Icarus, still considered a workhorse for DSMC applications. Michael and Steve began working on SPARTA about two and a half years ago — doing some of their brainstorming while walking on the Santa Fe Plaza during breaks at an international DSMC workshop Sandia hosts every two years.

“Michael is very, very good with DSMC physics and Steve Plimpton is very, very good at formulating problems so that large parallel machines can solve them. That combination has given us a parallel code with very sound physics that runs quickly,” manager Dan Rader (1513) says.

DSMC, invented in the 1960s by Graeme Bird when he was at the University of Sidney, takes a different approach from most codes that treat gases as a continuum rather than as individual molecules. Continuum codes solve partial differential equations based on such fundamentals as conservation of mass, momentum, and energy. Although it’s based on the same physical principles, DSMC’s molecular approach lacked formal mathematical proof of its soundness until 1992.

“Until then, people assumed it was kind of an approximate method that could do the job in regimes where continuum methods would clearly fail,” Michael says.

Sandia is largely interested in DSMC for two research areas where gas molecules are relatively far apart: re-entry vehicles, including the effects of flight through the outer reaches of the atmosphere, and micro-electrical-mechanical systems (MEMs) that have features at the micron and submicron scale. Examples of MEMS include chemical flow and pressure sensors, accelerometers, and transducers.

Senior manager Steve Kempka (5940), former senior manager in 1510, says the Gallis-Plimpton collaboration also has resulted in DSMC being used to simulate higher-density flows where molecules are relatively close to one another — something he never thought he’d see.

“DSMC will allow simulations free of the assumptions used in many other computational fluid dynamics methods. We hope it will let us explore the physics of turbulent flow in new ways. I believe we will see it being used to simulate re-entry vehicle flight to much greater depths in the atmosphere, with much more accurate descriptions of the flow,” he says. That includes the difficult phenomena of laminar, or streamline, flow transitioning to turbulent flow; ionization of the atmosphere flowing around a re-entry vehicle; and the wearing away of material as a space vehicle races toward Earth.

The chaotic nature of turbulence makes it difficult to investigate, but Michael says being able to use DSMC to study fluid mechanics at a more fundamental level may help to better understand the mechanisms of turbulence.

Parallel computing has broadened DSMC’s uses

Particle methods like DSMC saw relatively limited use in the past, but parallel computing has broadened their applicability.

“We can now look at problems that years ago were unthinkable, or go well into the continuum regime, when we think the relevant physics needs modeling at the molecular level,” says Michael.

“Compared to Icarus, SPARTA enables us to resolve finer scales of behavior in low-density fluids and study flows around more complex geometries,” Steve Plimpton says. “In its development, we also thought about future, faster computers with novel architectures, trying to design the code to work well both on today’s machines and tomorrow’s.”

With an open-source code, Sandia hopes to collaborate outside the Laboratories to improve the code and enable users to add their own new physics, chemistry, or collision models, Steve Plimpton says.

“It’s a win for Sandia because having additional developers expands the capabilities of a code more quickly and we get to use ideas that other people add to it and vice versa,” he says. “It also introduces us to users who may discover coding errors or wish to collaborate with us on new applications.”

The next frontier is to take advantage of DSMC to model and study flow physics — energy exchange and chemical reactions in colliding molecules — at a more fundamental level than possible with continuum codes, Michael says. “If you’re able to study things at that level, you are able to extract models that you can then use in continuum codes to study much bigger problems,” he says.


-- Sue Major Holmes

Back to top of page

Sandia report draws lessons learned from ‘perfect heists’ for national security

SANDIA RESEARCHERS Alex Roesler, left, Luke Purvis, and Jarret Lafleur, shown here inside a Bank of Italy vault in a historic Livermore building, studied 23 high-value heists that occurred in the last three decades for lessons learned that can be applied to designing complex security systems to protect vital national security assets.          (Photo by Dino Vournas)

by Patti Koning

On Feb. 17, 2003, the unthinkable happened at Belgium’s Antwerp Diamond Center. Thieves broke into its reputedly impenetrable vault and made off with hundreds of millions of dollars’ worth of diamonds, gold, cash, and other valuables.

Through years of meticulous planning, they got past police officers less than 200 feet away, access controls into the building, a combination-and-key-lock vault door, a magnetic seal on the vault door, and motion, infrared, light, and seismic detectors in the vault.

The Antwerp Diamond Center theft and other sophisticated, high-value heists show that motivated criminals can find ways to overcome every obstacle between them and their targets. Can the Energy and Defense departments, responsible for analyzing, designing, and implementing complex systems to protect vital national security assets, learn from security failures in the banking, art, and jewelry worlds?

Sandia systems analyst Jarret Lafleur (8118) set out two years ago to answer that question. “There are many insights to be gained from studying high-value heists and related crimes that could be applied to Sandia’s work in physical security,” he says. “Our work focuses on securing nuclear materials and other assets. Those kinds of attacks and threats are extremely rare, which is good, but give us very little historical information to draw upon.”

Compiling the crimes

He found there hadn’t been a comprehensive study of sophisticated and high-value heists in more than two decades. “When we dug into the details, we found several areas worthy of further study that could inform our approach to physical security,” he says. “Two examples are the roles of insiders in successful heists and the ways that redundancy in a security system can affect the behavior of humans in the loop.”

Using public information sources, Jarret chose 23 worldwide heists that occurred in the past three decades, notable for the value of assets stolen, innovation, and complexity. The thieves used kidnapping, violence, and the threat of violence, insiders both coerced and willing, and many forms of deception. Other key ingredients were patience, innovation, and meticulous planning.

Notable cases Jarret looked at included the Vastberga Helicopter Heist (Sweden, 2009) in which thieves descended from a helicopter into a cash depot by smashing through a skylight; the Isabella Stewart Gardner Museum Art Heist (United States, 1990) where burglars posed as police officers to deceive and subdue museum guards; and the Securitas Cash Depot Heist (Britain, 2006) that saw robbers abduct the manager, his wife, and their child to force him to let them into the depot and provide key details about its security.

Jarret, working with Luke Purvis (8119), manager of Sandia’s National Security Systems Analysis group, and Alex Roesler (5627), manager of the Assurance Technologies and Assessments group, compiled the results in a Heist Methods and Characteristics Database. They analyzed the results qualitatively and quantitatively to describe the range and diversity of criminal methods and identify characteristics that are common or uncommon in such high-value heists. The analysis focused on seven areas: defeated security measures and devices; deception methods; timing and target selection; weapons employed; resources and risk acceptance; insiders; and failures and mistakes.

Jarret, Luke, and Alex published the research in a report “The Perfect Heist: Recipes from Around the World” [SAND 2014-1790], which details the 23 crimes, their categorization, and lessons learned. Jarret also presented the “The Perfect Heist” to numerous audiences.

Deception, patience are common ingredients

While methods and implementation of the heists varied greatly, there were common factors. At least one form of deception was used in 21 of the heists, ranging from impersonating law enforcement to use of decoy vehicles to concealing surveillance equipment. Insiders — willing, unwitting, and coerced — played a role in the majority of cases. The Antwerp Diamond Center’s building manager even provided blueprints to the heist mastermind, thinking he was just another tenant. Jarret continues to delve into the concept of the coerced insider with the help of interns from the Air Force and Naval academies.

“I learned from this study that these thieves have a lot of patience. Most spent months and even years planning. They were very deliberate in how they defeated security measures and those methods were often very low-tech, like using hair spray to disable infrared sensors,” says Jarret. “In most of these heists, multiple security measures were defeated.”

Another finding is that weapons aren’t needed to steal a lot of money. Four of the top five heists, in terms of value, were weaponless.


-- Patti Koning

Back to top of page

Sandia brings home four regional tech transfer awards

Daniel Dedrick (8367), Sandia hydrogen program manager, and H2FIRST partner Catherine Dunwoody of the California Fuel Cell Partnership are working to install more fueling stations for hydrogen fuel cell cars.

by Nancy Salem

Sandia won four regional awards from the Federal Laboratory Consortium (FLC) for its work to develop and commercialize innovative technologies.

The Far West/Mid-Continent Region awards honored:

  • New Mexico Small Business Assistance (NMSBA) program as a Notable State and Local Government Collaboration;
  • Mantevo software, which recently won an R&D 100 Award, for Excellence in Technology Transfer;
  • Nano-Stabilized Enzymatic Membrane for CO2 Capture as a Notable Technology Development;
  • H2FIRST, an effort to build the number of fueling stations for hydrogen cell electric vehicles, as the Outstanding Regional Partnership, involving Sandia and the National Renewable Energy Laboratory, California Fuel Cell Partnership, California Governor’s Office of Business and Economic Development, and California Air Resources Board.

“Sandia is proud of our work in technology development, technology transfer, and partnerships,” says Jackie Kerby Moore, manager of Technology and Economic Development Dept. 7933 and Sandia’s representative to the FLC. “We are truly honored to be recognized, alongside our partners, for our work in these areas.”

Here are the four Sandia winners:

NMSBA: A helping hand to small business

Some 354 small businesses in 29 counties participated last year in NMSBA, a public-private partnership among Sandia, Los Alamos National Laboratory, and the state of New Mexico that connects small business owners with scientists and engineers who provide technical assistance. The program also contracts with the New Mexico Manufacturing Extension Partnership, University of New Mexico Management of Technology program at the Anderson School of Management, Arrowhead Center at New Mexico State University, and New Mexico Tech Department of Management. NMSBA provided $4.6 million worth of assistance to New Mexico small businesses last year.

“NMSBA has been bringing small businesses together with scientists and engineers from Sandia and Los Alamos national laboratories for 14 years. We are grateful to the principal investigators who work with New Mexico’s small businesses,” Jackie says. “Together they are implementing innovative ideas and stimulating our state’s economy.”

NMSBA was created in 2000 by the state legislature to bring national laboratory technology and expertise to small businesses in New Mexico, promoting economic development with an emphasis on rural areas. Since its inception, the program has provided 2,195 small businesses in all 33 New Mexico counties with more than $39 million worth of research hours and materials.

Mantevo: Next-generation computing

Mantevo is an integrated collection of small software programs, or miniapps, that model the performance of full-scale applications yet require code a fraction the size of the full application. They were designed to improve high-performance computing.

The Mantevo project, which grew out of Laboratory Directed Research and Development (LDRD) and won a 2013 R&D 100 award, pioneered the miniapp concept. Mantevo Suite 1.0 is the first integrated collection of full-featured miniapps.

Mantevo provides rapid exploration of the design space for new supercomputers and applications. It speeds research in today’s co-design model of collaborative computer development, where designers work together simultaneously on hardware and software. It is a critical tool that impacts business, science, and national security.

Sandia brought the Mantevo miniapps together and made them freely available as an open-source software package. Every major computer vendor and dozens of university research teams use Mantevo to design next generation systems and tools.

The work was done in collaboration with, among others, Los Alamos and Lawrence Livermore national laboratories and Santa Clara-based NVIDIA Corp.

“Things are changing rapidly in the computing industry,” says Mike Heroux (1426), co-lead on the project with Richard Barrett (1423). “It is very satisfying that the work is having an impact on the design and development of new computing systems and applications.”

CO2 Capture: Better way to keep emissions at bay

Electricity-generating plants, especially coal-fired, and other industrial activities that generate CO2 face new US Environmental Protection Agency regulations limiting emissions.

Nano-stabilized enzymatic membranes for CO2 capture provide a simple, compact, and more energy-efficient approach to capture than conventional methods. The process removes 90 percent of the CO2 in emitted gas mixtures and is expected to save the US coal industry alone $90 billion a year.

 Enzymatic membranes also separate CO2 from gas mixtures faster and more selectively than other membranes. The enzymatic membrane technology uses computational modeling to design and nanofabrication techniques to synthesize membranes with thin, narrow, and airtight liquid layers. By embedding an enzyme that converts CO2 gas into a more soluble form, the liquid-layered membrane efficiently captures CO2 from power plant flue gas. The fabrication process can be inexpensively scaled-up. The robust and inexpensive enzyme is already used at industrial scales.

The CO2 capture process evolved from Sandia’s LDRD program. Team members included Susan Rempe (8635), Juan Vanegas (8635), and Jeff Brinker (1000).

“With the growing concern about global warming caused by greenhouse gas emissions, there is an immediate and urgent need for efficient CO2 capture and reuse in other applications such as enhanced oil recovery,” Susan says.  “It is gratifying to be recognized by the FLC for our Sandia-UNM team’s efforts to provide a solution to CO2 capture that works within the framework of the US coal industry.”

H2FIRST: Places to refuel hydrogen cell electric cars

As hydrogen fuel cell vehicles continue to roll out in increasing numbers, the infrastructure for fueling them must expand as well. A project launched by DOE and led by Sandia and the National Renewable Energy Laboratory (NREL) will work in support of H2USA, the public private partnership introduced in 2013 by the Energy Department and industry stakeholders to address the challenge of hydrogen infrastructure.

Established by the DOE’s Fuel Cell Technologies Office in the Office of Energy Efficiency and Renewable Energy, the Hydrogen Fueling Infrastructure Research and Station Technology (H2FIRST) project will draw on existing and emerging core capabilities at the national labs to reduce the cost and time of new fueling station construction and improve the stations’ availability and reliability. The partners include several agencies from the state of California, widely regarded as the nation’s epicenter of zero-emission vehicles.

The goal is to accelerate and support the widespread deployment of hydrogen fuel cell electric vehicles. Toyota recently announced it will begin selling its fuel cell vehicle in 2015. General Motors and Honda have announced plans to jointly develop hydrogen fuel cell cars, and Hyundai will lease its Tucson Fuel Cell hydrogen-powered vehicle in California.

“The success of hydrogen fuel cell electric vehicles largely depends on more stations being available, including in neighborhoods and at work, so drivers can easily refuel,” says Daniel Dedrick (8367), hydrogen program manager at Sandia. “With H2FIRST, we’re definitely on the road to making that happen more quickly.”



-- Nancy Salem

Back to top of page

Download Lab News August 8, 2014 (PDF, 2MB)