Sandia leads in large-scale passive optical network

SENIOR ENGINEER Steve Gossage (9336) looks at fiber optics in a cable box that replaced heavier and bulkier copper cable for high-speed communications throughout much of Sandia. Fiber offers more capacity and is more reliable than copper. (Photo by Randy Montoya)

by Sue Major Holmes

It took a lot to get that little white cable box with the dancing green lights on the walls in offices throughout Sandia.

The years-long effort has made Sandia a pioneer in large-scale passive optical networks with the largest fiber optical local area network in the world. It reaches 265 buildings and 13,000 computer network ports and brings high-speed communication to some of the Labs’ most remote areas for the first time.

The conversion will save an estimated $20 million over five years through energy and other savings and not having to buy replacement equipment. The network will reduce energy costs by 65 percent once it’s fully operational.

Fiber offers far more capacity, is more secure and reliable, and is less expensive to maintain and operate than the traditional network using copper cables.

An optical local area network (LAN) gives people phone, data and video services using half-inch fiber optic cables made of 288 individual fibers, instead of the conventional 4-inch copper cables. Copper cables used to fill up underground conduits and required steel overhead racks of connecting cable, along with distribution rooms filled with separate frames for copper voice and data cables. The fiber distribution system uses only part of the conduit and needs only a 2- by 3-foot cable box.

“The frames go away, and the walls are bare and the tray empties,” says senior engineer Steve Gossage (9336), who has spent his 36-year career at Sandia in advanced information and network systems engineering.

Pushing the boundaries of speed

The national laboratory has always pushed for speed beyond the fastest transmission rate available, Steve says. “When people were working in much slower data rates, kilobit-type rates at short distances, we were trying to get 10 times the distance and 10 times the speed,” he says.

Sandia began looking at fiber optics early in the technology’s development because of its promise of higher bandwidth — greater communication speed — at longer distances. “A lot credit has goes to Rich Gay [8949],” says Len Napolitano, director of Computer Sciences and Information Systems Center 8900. “He was an early advocate decades ago for full optical networks at Sandia’s California site, and has demonstrated it could work at a site-wide scale.” 

The Labs started converting from copper in the 1980s, first installing then-emerging fiber optics in a single building and bumping that facility to megabit speeds. “Today we’re way past that. We’re at 10 gigabit-type rates and looking hard at 100,” Steve says.

After years of planning, Sandia completed a formal network plan in late 2008 and sought competitive bids the following year. Sandia selected Tellabs of Naperville, Ill., as the equipment vendor for the network, and Steve and his colleagues simultaneously began to jumpstart the deployment of the fiber infrastructure and set up a test lab to validate the performance of configurations for the equipment and various network functions. The technology began moving to desktops in 2011, and by the end of 2012, Sandia had converted more than 90 percent of bulky copper cable to a fiber optics LAN.

Sandia, which will spend about $15 million on the project, needs superb computing capability for the problems it tackles as part of its support for the mission for the National Nuclear Security Administration.

“Whether it’s a materials science problem or modeling an event, we need a lot of data and a lot of processing capability,” Steve says. “We need to be able to see it, we need to be able to view it, we need to be able to put teams together. This is a large laboratory, deeply stocked with scientists and engineers and test labs. For the analyses we get, the problems are not small and they’re not easy.”

Since its first experience with fiber optics, Sandia envisioned being able to use multiple wavelengths in a very high bandwidth single strand reaching the farthest tech areas. But decades ago, when Sandia began putting in single-mode fiber to desks and adding underground fiber capabilities, the technology wasn’t quite mature enough to take advantage of fiber optics’ inherent multiple wavelengths and speeds.

Waiting for technology to develop

Sandia continued to install the fiber optics cable foundation and waited as the technology developed, and moved quickly when commercial optical networks began deploying voice, data, and video to large collections of homes and offices.

“There weren’t that many unknowns for us because we had been thinking about ways to do this on a large scale for quite a while,” Steve says. “We had already thought through what this might mean to us, what it might mean to our lifecycle costs and where the investments would be, and we were already pretty comfortable with fiber and the technologies that go with it.”

Buildings with conventional copper LANs have separate networks for phones, computers, wireless, and so on. Fiber optics puts everything in a single network cable. That eliminates a large number of power-consuming switches and routers and makes the network simpler to operate and cheaper to install. Since it requires less space, energy and maintenance costs go down.

“As we research and deploy new technologies our main objectives are to enable the Labs’ mission, decrease life-cycle costs and if possible reduce our footprint on the environment. With the deployment of passive optical networks we have been able to meet and exceed all of these objectives,” says manager Jeremy Banks (9336).

Where a conventional LAN serving 900 customers requires a space the size of three double ovens, an optical network serving 8,000 requires a microwave oven-sized space. Where copper cable required Sandia to maintain and manage 600 separate switches in the field, optical LAN allows it to operate a data center in one building and simple, standard ports to offices. Because fiber optics reaches beyond the 100-meter radius that once was the standard from a wiring closet to a desktop, remote areas such as the National Solar Thermal Test Facility have high-speed communications for the first time.

The only copper wire for most of Sandia today is a short connection from the wall to the desktop. Everything behind the wall is fiber.

Going from copper to fiber

Moving away from copper wasn’t easy. It required new technology for the core communication system and made Sandia its own network provider, Steve says. He credited a central team of about 10 people across Sandia who worked together every day throughout 2011, plus sub-teams totaling about 40 people. The effort included engineering design, information technology, network systems, computing, facilities, security, and people in the field pulling cable and connecting ports.

“’Thank you’ is just not quite enough when you see people working that hard for that long to assist in change, because change is hard and worrisome and disquieting,” Steve says.

Sandia is recycling copper as it’s replaced, which keeps tons of valuable material out of a landfill. The estimated $80,000 for the copper will offset some of the fiber optics cost.

The Labs also must turn off hundreds of switches before it can fully realize the energy savings. That will take longer because it depends on such things as staffing, Steve says.

More change is possible

More change could be coming. A small trial is under way for voice-over-fiber — putting data and voice in one system rather than the two Sandia uses today. Testing shows Sandia can protect voice running through a congested circuit — what Steve calls “a Mother’s Day test,” when everyone calls at the same time. The Gigabit Passive Optical Network standard Sandia works with can dedicate part of the bandwidth and give priority to selected traffic such as voice. So calls would go through even with heavy competition from data.

Sandia also is working with a small number of researchers who need more bandwidth than they’re getting. The Labs’ needs are ahead of the market but it’s pushing for next-generation increases in speed, Steve says.

Communication speed improves every five to eight years. With copper, each improvement required replacing large, heavy bundles of jacketed cable to re-engineer them to perform at the new speed, he says. Fiber optical cable offers a bandwidth good for 25 years or more.

“We change the wavelength, we change the modulation rate, we don’t get back in the ceiling, we don’t get back in the customer’s office,” Steve says. “So our return on investment, our capital investment, our operational investment, the impact on our customers — everything gets better.”


-- Sue Major Holmes

Back to top of page

Sandia 'hears' Chelyabinsk meteor

BOLIDE DETECTED — This image shows the shortest path that infrasound waves from the meteor’s explosion propagated from Russia to receivers in the US. (Image data courtesy of Google earth and NASA)

by Heather Clark

 Darren Hart (5736) was busy reviewing data from the previous night’s test of infrasound sensors from a US station that monitors for nuclear detonations at Sandia’s Facility for Acceptance, Calibration and Testing (FACT) site. So when co-workers rushed in to tell him about a meteor the size of a bus that had exploded in a blaze of fire across the western Siberian sky, it was news to him.

“They came in the trailer and asked, ‘Did we pick up the event?’ and I said, ‘What event?’” Darren recalls. “I went over to the station and pulled up the viewer and lo and behold right in the middle of the screen was the event itself.”

Darren saw small disturbances in the atmosphere detected 6,200 miles (nearly 10,000 kilometers) away in the quiet southeast corner of Sandia Labs and about nine hours after the meteor’s explosion, called a bolide. The bolide occurred 12-15 miles above the Earth and released nearly 500 kilotons of energy. The meteor, which had an estimated mass of 10,000 tons, injured more than 1,000 people, mostly from breaking glass, according to NASA and media reports. It was the largest meteor reported since one hit Tunguska, Siberia, in 1908.

“Any time we can detect coherent signals by our infrasound array and see an event that the rest of the world is seeing, that's a pretty neat thing to be part of that signal capture,” Darren says. “From the distance, I was a little surprised to have picked it up, but then after hearing of the size, we weren’t as surprised as we initially were.”

Darren ran a simple analysis of the signals from an array of sensors at Sandia and determined the signal came from the north.

Kyle Jones, an infrasound geophysicist in Ground-Based Monitoring Research & Engineering Dept. 5736, is working on a more detailed analysis that could provide data for an international community of infrasound scientists.

Detecting pressure changes

The FACT site infrasound array detected the meteor’s signal at a frequency of less than 1 hertz by the time it reached Albuquerque. Infrasound is the study of sound waves at frequencies inaudible to humans, less than 20 hertz, Kyle says. For comparison, a bumblebee’s buzz is typically 150 hertz and humans hear in the range of 20 to 20,000 hertz.

At FACT, microbarometers — equipment first developed in the early 1900s that today are about the size of gallon milk jugs — are set up in arrays so that each sensor detects the atmospheric pressure change caused by the low frequency sound waves from the meteor’s explosion at slightly different times.

Calculating the varying arrival times here and at other infrasound sensors in the Southwest, Kyle uses geometry to detect the back azimuth, or the direction the sound originated. He says the sound waves detected here passed over the North Pole, which was the shortest distance for the pressure waves to travel.

Kyle also calculated the effects of thermospheric, stratospheric, and surface winds on the sound waves and atmospheric temperature changes that bounce the waves back to the ground multiple times during their journey. He calls this “ray tracing,” which recreates what happened to the sound wave as it was propagating away from the source and toward Sandia’s sensors.

“The fact that the FACT site detected this event at such a great distance and came within 200 kilometers of the source was very, very good,” Kyle says.

Kyle believes the Chelyabinsk bolide is the farthest event the FACT site has ever detected.

“For this type of event, this is probably the limit, but it depends on the winds and the path that the sound is going to take,” he says. “If the winds are favorable winds and the arrays are located at the right spot, it could carry much farther.”

While Sandia’s FACT site is not charged with monitoring nuclear explosions for treaty verification, it does have a research role in supporting the international community’s nuclear monitoring.

Kyle is currently part of a team that is developing software in a research capacity that he hopes both the US and international communities will utilize that uses near real-time wind profiles in its infrasound analysis of suspected nuclear explosions. After Kyle analyzed the wind’s effects on the Chelyabinsk bolide’s sound waves, he reduced the range of the possible location of the explosion by about 185 miles (about 300 kilometers).

‘Seismic is primary’

In the case of the recent meteor, the location was roughly known, but had the explosion happened where no one saw it, for example, over the ocean and unseen by ships, it would have been a mystery, Kyle says, so more accuracy in pinpointing the location would have been necessary.

“The international community uses infrasound as a secondary means of identification of suspected nuclear detonations. Seismic is primary,” Kyle says. “I would say infrasound is important because it's another piece of the puzzle.”

Infrasound can detect atmospheric detonations and the effects on the atmosphere from underground explosions, he says.

Infrasound also provides the United States with independent verification of what the international community is observing in its monitoring system and “it's a technology that we can share with them, so they can advance their understanding and they can help us advance ours,” Kyle says.

The FACT site is being expanded from a few acres to sensors across 400 acres and Sandia — with help from DOE, NNSA, the State Department, and KAFB — is helping form a network of infrasound arrays in New Mexico, Arizona, Nevada, Utah and California, called the Southwest US Seismo-Acoustic Network (SUSSAN), which will provide more data to improve the accuracy of autonomously locating all types of infrasound events, both natural and man-made.

-- Heather Clark

Back to top of page

Cyber Engineering Research Laboratory opens

THE TRUTH AS I SEE IT — Sen. Tom Udall presents his views on the challenges facing the US in cyberspace, and what Sandians and CERL can do to help secure that useful but increasingly threatened arena. To the right of the senator are NNSA official Dimitri Kusnezov, Albuquerque Mayor Richard Berry, and UNM Research VP McGraw.

by Neal Singer

 An unusual urgency underlay the brief speeches noting the formal opening of Sandia’s Cyber Engineering Research Laboratory (CERL) on Feb. 19.

This was possibly because of a warning of “malicious cyber activity” released the previous day by the FBI and the Department of Homeland Security, and the growing flood of news releases from large institutions announcing — sometimes admitting — they had been cyberhacked. 

The building — located in Sandia’s Research Park — is expected to serve as a venue to bring together expertise from across Sandia, as well as from universities and businesses, to develop innovative solutions against the increasingly serious challenges posed by hackers and cybercriminals to individuals, business, and government.

US Sen. Tom Udall, D-N.M., mentioned Winston Churchill’s book While England Slept, which in 1938 criticized the English government’s lack of preparation against the threat from Nazi Germany. Said Udall, “Cyberthreat is not one of guns and tanks but we need to take it seriously. . . . The threat is real to . . . our water systems, oil pipelines, hospital systems . . . and we should bring justice to those who would do us harm. CERL is a crucial part of our defenses.”

Challenge can't be taken on alone

Said Sandia President Paul Hommert, “[Cybercrime] can’t be tackled alone. The public and private worlds must combine efforts to work as a team.” Sandia’s cyberexpertise, he said, is rooted in its nuclear weapons history.

He mentioned Sandia’s Center for Cyberdefenders’ student internship program (, which has worked with more than 300 students in the past decade to hone the skills of next-generation cyber workers.

Other CERL projects include marrying algorithms and data in attempts to prevent adversaries from penetrating emails or damaging websites.

US Rep. Ben Ray Lujan, D-N.M.-3rd, mentioned that “[cyberdefense] is critically important to our economy. Work at Sandia and Los Alamos national labs should lead to partnerships with private businesses.” In terms of security, “personal information taken and used in some way, from an ATM machine or anywhere else, can allow someone from around the world to get into something personal [of a citizen’s here]. . . . People in Virginia [at security agencies] seem to have connected the dots and released . . . information about the current threats.”

NNSA official Dimitri Kusnezov said “the need for secrecy [has ranged historically] from clay tablets and cuneiform to today's complex protocols. . . . Our cybersecurity needs will not recede in time but only get greater as data complexity gets greater. . . . There is no scientific silver bullet. The key is to train our people to be more aware, smarter, building in as many safeguards as we can, codeveloped with technology. Centers like this can forward these steps.”

Albuquerque Mayor Richard Berry said he and the state’s congressional delegation were on the same team in supporting the work at CERL. He mentioned the takeover of four TV stations by attackers jokingly advertising “the zombie apocalypse” was not funny in what it said about communications security.

Adversaries getting more sophisticated

Peter Ungaro, president and CEO of Cray Computing, said “Cyber security is one of the largest threats out there today. . . . The vast amount of digital data is growing at an exponential rate — every two days, there’s more data created than from the dawn of civilization to 2003. Our hacker adversaries are getting more sophisticated in using data against us.”

Ungaro, who has made no secret of his admiration for Sandia in helping create what he termed “the most successful family of supercomputers ever built [based on the Sandia/Cray Red Storm supercomputer],” advocated  working together at CERL  to “develop a technical roadmap to take problems currently intractable and solve those to make them broadly applicable across a wide variety of frameworks.”

UNM research VP John McGraw, who advocated “strengthening the interest and intent of UNM colleagues to create new research ties [with Sandia] in energy, security and water,” also said that “Sandia’s unique mission is to protect the public against vulnerabilities not recognized by the public.”

Rob Leland, director of Computing Research Center 1400, said of the gathering, “I’m very touched by the turnout and by the excitement.” Just as the development of the laminar flow cleanroom ushered in a revolution in microelectronics, he said, “There’s the potential for us to do something similar in the cyber world and that CERL will play a key role in bringing that about.”

Three CERL demonstrations following the talks included:

  • A table-size interactive horizontal display on which large amounts of email traffic were represented. The screen demonstrated programs built to distill anomalies — harmful messages that are not what they seem, the sharks, so to speak, in the water — before they do damage;
  • Students wearing electroencephalograph (EEG) caps who saw signatures of their brain activity as they exercised various computational skills — an effort to improve the human element of the cyber equation to ultimately train better cyber defenders, and
  • Teams of students from New Mexico Tech, University of New Mexico, and local high schools who competed in a virtual cyber exercise to solve digital clues and catch a “bad guy.”
  • Duane Dimos, acting VP for Science and Technology, served as master of ceremonies.

“I’d like to mention our thanks to the large Sandia team from across the Labs who did significant work to help the conference go smoothly,” said Rob.

CERL is part of Sandia’s Cyber Engineering Research Institute (CERI), which also includes Cyber Technology Research Lab (CTRL) in Livermore, Calif., and industrial and academic members. The institute’s areas of interest include cyber data analysis, cyber modeling and simulation, cognition and human performance, and trusted systems.


-- Neal Singer

Back to top of page

Download Lab News March 8, 2013 (PDF, 2MB)