Why cell phones failed in PG&E outages, and how to prevent a repeat

November 04, 2019

As the lights flickered out and wildfires flared, PG&E’s blackouts also cut off thousands of Californians from cell phone service, leaving them unable to get emergency alerts or call 911.

It exposed a troubling gap in the state’s readiness for mass outages that could, according to PG&E, keep happening for a decade.

And it’s left regulators scrambling to find a fix — though it will be difficult.

Neither California nor the federal government requires cell phone towers to have backup power, even though network service is a critical part of modern life.

Instead, maintaining service is left up to cell phone companies, which have generators lasting days at some sites but batteries which can survive just a few hours at others. When those run out, they must send trucks to refuel or install generators, at times when fires may cut off roads, blackouts darken traffic lights and their own outages hamper communication.

Companies said their personnel worked around the clock to put in place hundreds of generators during PG&E’s unprecedented and fast-changing power outages, but in some cases they couldn’t access sites because of the location — on top of buildings or in fire evacuation zones. They’re pledging to prevent future problems. But regulators, politicians and emergency response agencies are pushing for stricter rules to protect public safety.

“Reaching 911 in an emergency can be the difference between life and death,” Sen. Kamala Harris said in a statement following a letter to companies last week demanding answers for their failures.

On Friday, nearly two dozen California members of the House of Representatives called for a hearing on telecommunications during emergencies.

“Our telecommunications infrastructure has failed many times when the public and public safety officials needed it most,” they wrote.

Jessica Rosenworcel, a commissioner with the Federal Communications Commission, criticized her agency for what she said was loosening regulations.

“This is a problem that demands investigation and solutions,” she said in a statement. “Mother Nature’s wrath is sure to visit us again.”

An estimated 80% of 911 calls are made with cell phones, and public agencies depend on phone notifications to order evacuations. During power outages, some without service weren’t getting alerts or couldn’t load websites, forcing agencies to scramble to reach people.

Internet-based landlines also conked out in the outages, though traditional copper-wire landlines worked.

The carriers

How companies prepared for and performed in PG&E’s fire-safety shut-offs:

AT&T

3% of cell sites went down during outages. 78% of sites have permanent generators that can run for 72 to 120 hours on a single fuel tank. 12% have batteries lasting a minimum of four hours if the site has no permanent generator and two hours if the site has a fixed generator.

Verizon

3% of service network didn’t function during outages. Larger sites have up to eight hours of battery backup. The majority have permanent generators that can run for 24 to 72 hours on a single fuel tank. The vast majority of sites without can be served with portable generators but a “discrete portion” of the network can’t be accessed.

T-Mobile

At the lowest point during the outages, 9% of cell sites in the Bay Area and Sacramento regions were down, but that improved by the end of last week. The company said its generators generally can run for 72 hours.

Sources: AT&T, Verizon and T-Mobile

The California Office of Emergency Services worked on a contingency plan to evacuate residents door to door, through loudspeakers on vehicles or helicopters, according to Brian Ferguson, a spokesman.

After hearing reports of failed cell service for emergency alerts, Brenton Schneider, a spokesman for the Marin County Sheriff’s Office, said the agency was looking into using radio broadcasts as an alternative.

Schneider said officials in Marin, where PG&E cut power to 99% of the county Saturday and cell phone failures were pronounced, were “lobbying with AT&T and other providers to provide adequate backup power at sites” and will meet with cell carriers this week to press the issue.

Cell phone problems during California crises aren’t new. In the 2017 Tubbs Fire, Sonoma County received criticism for not pushing out emergency notifications earlier. During the Mendocino Complex fire last year, fire officials said Verizon slowed Santa Clara County firefighters’ data speeds after the department reached its plan limit. And in the Camp Fire, communication was crippled when cell towers went down.

In anticipation of the PG&E shut-offs, the five major carriers — AT&T, Verizon, T-Mobile, Sprint and U.S. Cellular — told the FCC in September they were prepared to keep up power with backup batteries, permanent generators and mobile cell sites unless wildfire damaged infrastructure, which companies confirmed wasn’t the issue last month. Ferguson, the emergency office’s spokesman, said he heard from providers that “they weren’t concerned.” During the outage, backup power either proved unavailable or didn’t last long enough, he said.

As lights shut off, so did up to 3% of cell sites statewide, FCC reports revealed. Hundreds of thousands of customers lost connections to phones, internet and TV. In the hardest-hit county, Marin, 57% of towers were down on Monday, in a crucial period between two periods of high wind. Even some landlines went dead.

“Cell service, for the first few days of this, was the one thing we had, and then when it started failing, it really got ugly,” Rep. Jared Huffman, D-San Rafael, said. “We’ve got to get this fixed.”

The California Public Utilities Commission is very concerned about network resiliency and is actively investigating the issue, spokesman Christopher Chow said in an email Friday. When Gov. Gavin Newsom declared an emergency as fires swept the state, the commission triggered its requirements for companies to deploy mobile cell sites, provide charging stations, Wi-Fi and phones for evacuees, and urged them to defer payment and lift data caps, which they did, Chow said.

But there are limits to the commission’s power. Current law bars the state regulators from overseeing internet-based phone services, which includes some newer landlines. That law will end in 2020, freeing up the commission to regulate those services, Chow said.

Legal filings with the state show that the industry, although stressing how vital service is, has challenged the agency’s authority to impose backup-power requirements especially on internet-powered phones, called VoIP.

AT&T spokesman Jim Kimberly said in an email the VoIP Coalition already voluntarily offers support to customers in the wake of disasters and objected that the mandate was outside PUC’s jurisdiction.

The state utilities commission’s Public Advocates Office pushed the agency earlier this year to require carriers to harden infrastructure — like installing permanent generators at all cell towers.

“These power shut-offs have shown how vulnerable they are,” Ana Maria Johnson, a program manager in the office, said. “The companies need to be held accountable to ensure, through requirements, not voluntary, that there is sufficient power in place so that consumers are able to use those essential services to get those alerts, to communicate with loved ones, and to follow info on what they need to do.”

Companies said they were affected, like all customers, by PG&E’s unprecedented power outages. Comcast and Frontier, which provide internet, TV and phone services through wired networks, said backup systems can’t last indefinitely and service disruptions “may be unavoidable.”

State Sen. Mike McGuire, D-Healdsburg, introduced a bill this year that would require cell towers in zones with the highest fire risk to have at least 48 hours of backup power. But after these outages, he’s reconsidering whether that’s long enough, he said.

Assemblyman Marc Levine, D-San Rafael, who sponsored a bill this year to make sure first responders have full access to data, said he believes communities should find a solution, not necessarily mandate it. He said he often hears neighborhoods don’t want additional cell towers or are concerned about the environmental impact of storing fuel for backup power.

Levine placed primary blame on PG&E for pulling the plug on millions.

“This can’t be normal. PG&E must be held accountable,” he said. “This is a lesson where cell service providers had dead spots, where they need to shore up. It’s a collaborative process to identify places when PG&E is causing this problem.”

Bay Area residents fear that cell phone companies’ poor performances during predictable, planned outages is a precursor to bigger failures.

“The widespread power outage is a miniature rehearsal for the next big earthquake,” Davis Carniglia, a Marin resident who lost power and cell service for four days, said. The retired business executive is also a former volunteer firefighter and served as a member of the Marin Disaster Council.

“The idea that there are cell towers out there that only have batteries is nuts,” Carniglia said.

Companies have pledged to prevent future problems.

AT&T spokesman Jim Kimberly said in an email the company spent more than $8.2 billion during the past three years to maintain and update its California network.

“Following any large-scale event, we strive to make improvements where needed and remain focused on serving our customers,” he said.

T-Mobile spokesman Joel Rushing said in an email the company “did our best to minimize interruptions for our customers and will continue to evaluate ways to keep supporting our customers during these events.”

In a letter to the state emergency office last week, Verizon said, “We recognize this is not an isolated event and we are preparing for future events even as the current (power shut-off) winds down.”

Because PG&E has warned that shut-offs as a fire-prevention measure will continue, some are looking to new technology for answers.

The California Energy Commission gave a grant this year to Caban Systems of Burlingame to produce solar-powered batteries for telecommunications companies. It already sells such systems in Latin America.

CEO Alexandra Rasch said the company is currently in talks with major cell carriers in California.


By:  Melissa Moench
Source: San Francisco Chronicle