WASHINGTON — Investigators traced Friday’s nationwide ground stop to a newly installed router at a remote radio site in southern Arkansas, after the machine began flooding the Federal Aviation Administration’s aging communications network with data, The Wall Street Journal reported Saturday.
The first sign of trouble came at the FAA’s Indianapolis center, which handles high-altitude flights over a large part of the middle of the country. Static crept into radio calls. Pilots could not always hear controllers clearly, and some transmissions were cut off.
Troubleshooters started in Indianapolis. Their search soon moved to two other places, people familiar with the matter told the Journal: an FAA radio site in El Dorado, Ark., and a radar site in Amarillo, Texas.
Network equipment at the Arkansas site, and possibly at the Texas site, was sending far more data than the system could handle.
“We saw two places that were actually pinging our network at a very high rate, a thousand connections over every four to five seconds,”
Those pings moved through the FAA Telecommunications Infrastructure, known as FTI. The network is a mix of old copper wires and newer fiber-optic cables. It carries radio transmissions, radar data and flight information between controllers and pilots across the country.
Defense contractor L3Harris Technologies oversees the system. The company installed the new Arkansas router not long before it began sending out a high volume of data on Friday afternoon, the Journal reported.
The router was made by Juniper Networks, part of Hewlett Packard Enterprise. About 60 Juniper routers are in use in the FAA network, a government official familiar with the system told the Journal, and the others were working normally.
A flood that reached 17 centers
Data from the Arkansas router, and from equipment in Texas, overwhelmed the network. FAA Administrator Bryan Bedford said 17 of the FAA’s 22 centers that oversee high-altitude flights had communications disruptions.
That was enough to stop departures. The FAA ordered a nationwide ground stop on Friday, Nov. 13, after trouble was first noticed at 3:36 p.m. ET. The stop took effect at 3:54 p.m. and was lifted at 8:15 p.m., a halt of more than four hours.
It was the longest nationwide ground stop since the Sept. 11, 2001, attacks. The only other such stop since then came in January 2023, when a failure in the pilot notification system paused departures for about 90 minutes.
Airlines canceled nearly 10,000 flights on Friday and delayed more than 7,000 others, Reuters reported. By Saturday, more than 20,000 flights had been disrupted since Friday.
Passengers spent Friday evening rebooking flights, renting cars and looking for hotel rooms while FAA crews worked to find the source of the failure.
Two sites cut off, and the network steadied
The problems stopped when the FAA cut off the Arkansas and Texas equipment from the wider network.
“We disabled both of those right away and were able to stabilize our network,” Guy said.
What officials still do not know is why the router behaved that way. Bedford put the open questions plainly on Saturday.
“Was it a faulty installation? Was it a faulty configuration? Or was it just a bad piece of router technology that we put into this system that propagated these errors?” Bedford said, noting that the FAA is still investigating.
Officials did find one feature that appeared to be part of the problem. A government official described it as a broadcasting feature that sent data more widely than needed, the Journal reported. The FAA has since turned that feature off on all of its Juniper routers.
HPE took a different view of its equipment. A spokesman said the Juniper devices appeared to have been working as intended.
“They did not fail,” he said.
L3Harris said it takes the ground stop and its effect on travelers, airlines and air traffic operations seriously, and that it is supporting the FAA’s investigation.
No sign of an attack
Transportation Secretary Sean Duffy said investigators had found no evidence of a malicious cause. The focus, he said, was on a mechanical problem with the router itself or on what he called “incompetence” by L3Harris.
That question — a faulty router, or a router that was installed or set up badly — is still unanswered.
The failure also revived a warning issued three years ago. A 2023 outside review said growing outages and too little money for overhauls could push the system’s risk to unacceptable levels, the Journal reported.
Bedford said the FAA spends hundreds of millions of dollars each year just maintaining its old telecommunications systems. L3Harris has been the prime contractor for FTI since 2002, and the company says it has converted about 70% of the copper-based network to fiber.
Work on the system is usually done overnight in busy areas. In remote places, officials had assumed daytime work would not affect many flights. Duffy said that assumption will now be re-examined, and officials will re-evaluate when such work is done.
He also warned that Friday may not be the last disruption of its kind. Until all of the copper wiring is replaced, which is not expected until the end of 2028, more problems could follow.
“This was an FAA system that didn’t meet the moment. Our contractor didn’t meet the moment. Or the equipment didn’t meet the moment,” Duffy said.
Virginia Times reported Saturday that Duffy had first pointed to a faulty router installed during copper-to-fiber work as the likely cause. The Journal’s account adds the two sites where investigators focused, the rate at which they hit the network, and the router feature the FAA has now disabled across the fleet.
Also Read: FAA Orders Nationwide Ground Stop Over Communications Issue