Sandia Benchmarks Progress Toward Useful Quantum Computing
Sandia National Laboratories’ Quantum Performance Lab created a new quantum universal operation performance system (QUOPS) to benchmark and measure quantum computer performance. The QUOPS framework was designed to overcome key limitations in existing benchmarking protocols and enable system-level assessments of quantum computational capability. QUOPS is the first tool designed to enable the quantum computing community to track progress toward utility scale quantum computing.
Figure 1. The Quantum Universal Operation Performance System (QUOPS) enables stakeholders to track progress toward useful quantum computation, or “quantum utility”. A particular quantum computer’s capabilities are described by a capability region describing the programs it can run successfully. Its QUOPS score, a simple metric of performance, is the size of the largest program it can run successfully within a cone that defines “computationally-relevant” program shapes along the road to the first achievement of quantum utility (which can be predicted only roughly).
Stakeholders, including vendors, customers, governments, and investors can use QUOPS to measure, track, and compare the computational power of any quantum computer, whether it operates on physical qubits or fault-tolerant error-corrected ones. Google, IBM, and Quantinuum processors have already been benchmarked, with co-authors Quantinuum contributing the hardware demonstrations (including the first benchmarking of a fault-tolerant processor) and NVIDIA contributing an open-source implementation on its CUDA-Q platform. QUOPS is freely available today in CUDA-Q, Quantinuum’s Guppy, pytket, and Sandia’s pyGSTi.
Most existing benchmarking tools focus on the performance of components like qubits and logic gates. While these component metrics are valuable, they do not provide a holistic view of system level performance. Existing system level benchmarks like quantum volume were designed in an earlier era, and do not scale well to the large fault-tolerant machines that will enable utility-scale computing. These tools were sufficient in the Noisy Intermediate-Scale Quantum (NISQ) era, but the coming era of fault tolerant quantum computers and complex logical architectures will require new techniques that can account for the architectural tradeoffs inherent to fault tolerance.
The QUOPS framework establishes two primary metrics. QUOPS score defines an objectively measurable, cross platform “unit” of successfully executable quantum computation, while QUOPS rate captures the speed at which a system can execute those units. These metrics provide the foundation for tracking the growth in computational power of increasingly advanced quantum computing systems. The QUOPS framework is intentionally modular and extensible, enabling it to adapt to innovations in quantum algorithms, architecture, and hardware over the coming years.
At Sandia, measuring progress toward utility scale quantum computing is vital to ongoing research. Quantum computing is a top priority for the US Department of Energy, and as one of DOE’s national labs, Sandia is preparing for a future where utility scale quantum computers directly advance the nation’s most challenging scientific and national security problems. That future is coming into clearer focus. A recent executive order announced DOE’s pursuit of a Quantum Computer for Application Development and Discovery Science (QC ADDS), and DOE’s blueprint for quantum supercomputing laid out a path toward fault tolerant, utility scale systems.
While utility-scale quantum computing may still be five to ten years away, to deliver on DOE’s mission, Sandia researchers are exploring whether today’s hardware, tomorrow’s architectures, and emerging software ecosystems are truly moving toward machines capable of solving classically intractable problems. Tracking that progress requires transparent, cross platform metrics that reveal how quantum computers are improving—not just in isolated benchmarks, but in their ability to execute the kinds of computationally relevant programs needed for real scientific discovery.
The Sandia-led team demonstrated how QUOPS can be used to rigorously assess current system capabilities and project progress towards utility-scale capability. The QUOPS framework provides a rigorous way to track progress, ensure the nation is ready for the first generation of utility scale quantum computers, and is positioned to translate that capability into mission impact as soon as it arrives.
Assessing current system capabilities
The QUOPS framework describes a computer’s computational power using a capability region—a graphical representation of the computational “tasks” a system can execute reliably.
Figure 2. Capability regions describe a quantum computer’s computational power in QUOPS. A computer’s capability region describes all the programs or circuits, organized by their width (# of qubits) and size (# of gates), that it can run. Error mitigation can extend a machine’s capability moderately, at a fairly drastic cost in throughput (QUOPS per second).
The capability region for a machine is established by executing quantum circuits (programs) of varying sizes and recording which ones it completes successfully. A machine’s capability plot shows graphically which sizes and shapes of circuits it can reliably execute, providing a clear picture of its true computational reach.
Figure 3. To map out a quantum computer’s capability region, researchers run QUOPS circuits of various widths and sizes to see which ones it can execute correctly. QUOPS circuits are specified using an “intermediate representation” – arbitrary-angle single-qubit rotations and controlled-NOT gates between arbitrary pairs of qubits – that’s fairly concrete, but can be compiled onto any specific machine’s native operations and connectivity.
Projecting and tracking progress to utility-scale capability
Scale: The QUOPS framework can be used to quantify the gap between today’s state of the art quantum hardware and utility scale target workload requirements. State-of-the-art quantum algorithms research has established resource estimates for utility-scale challenge problems. These estimates can be translated to the QUOPS framework to define QUOPS target workload requirements.
Sandia researchers evaluated the number of QUOPS required for key national security workloads. The best-known algorithms to implement these capabilities will require 270 to 370 million QUOPS, far beyond current hardware capabilities. To characterize a system’s capability, the framework introduces the QUOPS score, defined as the size of the largest computationally relevant program it can execute reliably. Sandia and its partners report experimental measurements of QUOPS scores for systems from Quantinuum, Google, and IBM. As expected, these scores—ranging from 216 to 1,824 QUOPS—are still well below the requirements of utility scale workloads.
Figure 4. Measured QUOPS scores and capabilities for three cutting-edge quantum processors.
Sandia also used the new framework to measure the capability of a prototype fault tolerant architecture that executes computations on up to 8 logical qubits implemented on the 98 qubit trapped ion Helios 1 platform. The fault-tolerant processor achieved a QUOPS score of 40, which — while still trailing that of leading physical-qubit processors — demonstrates that fault-tolerant architectures are rapidly narrowing the gap.
Looking ahead
As the nation prepares for the first generation of utility scale quantum computers, the QUOPS framework establishes a shared, cross platform language that enables government, industry, and research communities to communicate consistently about quantum capability and to track measurable progress toward utility scale systems. By incorporating compiler overhead and supporting both physical and logical architectures, QUOPS creates a uniform basis for comparing diverse hardware platforms and monitoring long term trends— including key milestones such as when logical qubit performance surpasses physical qubit performance. Its modular, extensible design ensures that new metrics, workloads, and architectural paradigms can be integrated as the field evolves, providing a stable foundation for an assessment tool ecosystem that matures alongside emerging fault tolerant technologies. In this way, QUOPS offers a durable, data driven mechanism for projecting capability trajectories and for coordinating progress across the quantum computing community as it advances toward utility scale performance.
