Autonomous drones compete, demonstrate national security uses

The rules of tag are simple enough for a playground: catch or be caught.

But when the players are autonomous drones, the game becomes a test of machine learning, split-second decision-making and cooperative strategy.

Researchers at Sandia recently used reinforcement learning, a kind of machine learning, to teach drones how to play a strategic version of the game. The work could help advance algorithms for autonomous systems that need to pursue, evade or coordinate in rapidly changing environments, including future systems designed to defend critical facilities from hostile drones or other national security uses.

The project is part of a larger effort called AutonomyNM that uses machine learning and artificial intelligence to push the boundaries of what is possible in autonomy. Unlike remote-controlled drones, autonomous drones use algorithms to decide how to respond to a scenario, avoid obstacles and achieve a defined objective. Autonomy has the potential to enable more rapid responses and reduce human error in drone swarms.

The team’s work, including how the researchers trained the drones and compared their performance with other strategies, was recently published as a conference paper for the 2026 IEEE International Conference on Robotics and Automation.

“We have expertise in reinforcement learning algorithms,” said Spencer Jensen, Sandia computer scientist and lead researcher on the project. “Reinforcement learning is just beginning to be used in commercial robotics. The really nice thing about reinforcement learning is you can ignore a lot of extremely complex math — that’s probably slightly wrong anyway — and just get a best-effort algorithm that is going to be more flexible in rapidly changing scenarios.”

Reinforcement learning: Like training a puppy

In Sandia’s version of tag, two pursuer drones protected a safe “base” while two cooperating evader drones tried to reach it. The evaders were trained to work together to cause the pursuers to crash into each other, allowing the evaders to reach the base without being tagged — or, in this case, crashed into — by the pursuers, Spencer said.

The researchers used reinforcement learning to teach the evading drones how to cooperate. Reinforcement learning, like training a puppy to sit on command, uses virtual rewards to guide the algorithm toward the desired behavior.

“The challenge with reinforcement learning is you have to do a lot of reward shaping to get the behavior you want,” said Christian Llanes, a Sandia intern on the project and robotics doctoral student at the Georgia Institute of Technology. “Reward shaping is a twofold problem where you determine what behaviors you want to reward or penalize and how much reward or penalty you want to give.”

Christian started working on the project in fall 2023 after completing a summer internship on a similar project. He used reinforcement learning to train the evading drones’ guidance module in computer simulations. The guidance module dictates a path for the drone’s control module to follow.

For the evading drones, reinforcement learning offered a way to develop more flexible behavior without programming every possible move.

“Reinforcement learning is a simulation-based guidance technique,” Spencer said. “We really like reinforcement learning because that behavior can be very hard to optimally program but is something that reinforcement learning can just learn how to do optimally. It makes the complex simple.”

The pursuers were not trained and instead followed a more straightforward algorithm called proportional navigation, which simply seeks to intercept its target, Spencer said.

Going from a computer to the real world

Once the evading drones’ guidance module was fully trained, the team took the algorithm from simulation to the real world and tested it on small, inexpensive drones.

A major challenge was the difference between how an algorithm performs in a computer model and how it behaves on physical hardware.

“With reinforcement learning, one of the common problems is the simulation-to-reality gap,” Spencer said. “Because this is a simulation-based guidance method, it often doesn’t work very well in hardware because of all these unmodeled dynamics.”

The team adjusted for this by studying the small, inexpensive commercial off-the-shelf quadrotor drones more fully at the beginning of the project and better modeling them in the simulation, he said. However, some issues — such as how a draining battery affects rotor speed and how one drone can create wind that affects another — are hard to fully model. There also are many small communication delays in the real world that do not exist in simulations, Christian added.

The team used inexpensive drones that are about 3.5 inches wide and weigh less than a chocolate bar because they can crash into each other or drop out of the sky without ill effects beyond an occasional inexpensive plastic piece snapping off, Spencer said. Repairs are much faster and easier than on larger drones, allowing the team to test improved algorithms rapidly, he added.

The drone lab used for testing has a state-of-the-art infrared motion-capture system, similar to those used in Hollywood movies, to track the position of the drones with high precision. The facility, located in the Center for Advanced Manufacturing and Innovation, or CAMINO, allows Bluetooth-enabled drones to be tested, streamlines internships and facilitates research partnerships with universities and local businesses. 

“The high bay in CAMINO allows us to bridge the gap between early-stage research and high-consequence hardware,” Spencer said. “We can do low-cost testing in this facility. A lot of the work we’re doing is trying to bridge the simulation-to-reality gap. Our goal is to work out some of the bugs on low-cost hardware before we move on to high-cost, high-consequence hardware.”

This research project was funded by Sandia’s Laboratory Directed Research and Development program.