Emergency put communications to the test (NYC, 9-11)

greenspun.com : LUSENET : Grassroots Information Coordination Center (GICC) : One Thread

Headline: An Unimaginable Emergency Put Communications to the Test

Source: New York Times, 20 September 2001

URL: http://www.nytimes.com/2001/09/20/technology/circuits/20INFR.html

On any given day, Dave Johnson knows far more than most people about the many phone calls that are crisscrossing the United States. Mr. Johnson, a spokesman for AT&T, works in the company's global network operations center in Bedminster, N.J., where he and his colleagues monitor 141 vast projection screens that provide minute-by-minute details on the state of the country's largest long-distance system. By watching computerized maps of the United States, they can tell in an instant whether there are any jams in long- distance traffic.

On an ordinary day, Mr. Johnson said, those maps are blissfully blank, showing no unusual activity. But on Tuesday, Sept. 11, as soon as Mr. Johnson and his colleagues saw the first television reports of the terrorist attacks on New York and Washington, they knew that this would not be an ordinary day.

"We all looked at each other and said, here we go," Mr. Johnson said. Within minutes, the screens lighted up, showing tens of millions of calls many from worried relatives and friends that threatened to clog the system. "What started as a quite peaceful day," he said, "we instantly knew was not going to be quiet and peaceful any longer."

In Manhattan, the threat to the phone system was not just traffic it was physical damage. The collapse of the World Trade Center crippled many of the connections that downtown Manhattan depended on, threatening crucial links for the police and emergency crews. Cellular sites were knocked out. A switching office for local service was badly damaged by falling debris and burst water pipes. Fiber-optic transport equipment was crushed. Power failures cut off high- speed Internet service for many companies across the city.

By day's end, the telecommunications system in the country and particularly in New York had experienced what was probably the biggest test ever of its ability to withstand a physical attack and a national emergency.

So far, the results are encouraging. Several companies trucked in temporary cellular towers. The 911 system was never disrupted. Long-distance lines were kept open for New Yorkers who were calling out. For most people, the Internet was slowed but intact.

Bruised and battered, the communications infrastructure persevered, but not before providing lessons about how networks adapt to such extreme conditions and what more can be done to preserve them in emergencies.

Verizon had to deal primarily with the effects of sheer destruction. The company's office at 140 West Street was a central switching office for local telephone traffic. Five stories of debris from 7 World Trade Center, which collapsed Tuesday afternoon, fell against the building, which was also hit by a massive steel girder from the North Tower. The basement is flooded with water from shattered pipes, rendering the backup power generators useless.

The loss of all service at 140 West Street left as many as 175,000 customers in the area unable to make or receive calls. It also affected companies that used as many as 3.5 million lines that carry Internet traffic and private financial information through Lower Manhattan. Those lines, Verizon says, were either damaged or were attached to switches that have no power. Several companies had to use alternative communications channels to provide Internet access to their employees or subscribers.

"I suspect every business in Manhattan is in some way affected by the loss of communication," said Ivan Seidenberg, co-chief executive of Verizon. "Lower Manhattan is a pretty influential place in the entire scheme of life."

Among the companies affected was Earthlink, which was unable to provide high-speed Internet service to more than 7,000 subscribers until Friday. The effect rippled out to Washington, because Earthlink had been channeling some Washington subscribers through the New York center to balance its load. Kevin Brand, the company's vice president for network operations, said that to solve the problem the company rerouted traffic through several points in the Northeast that are controlled by one of its partner companies.

Failures of backup power also hit several other network facilities around downtown Manhattan. AT&T was unable to run telephone-switching equipment in the World Financial Center because of power losses.

Many cables in the area are also in bad shape. When water mains broke after buildings collapsed, the water seeped into cables that have since short-circuited, Verizon officials said. AT&T officials say they are certain that they lost several pieces of sophisticated equipment in the basement of the World Trade Center that were used to transport data over fiber-optic cables.

The phone system's losses were human as well. A half-dozen Verizon workers are assumed to have perished in the trade center's destruction, among them two technicians who were in the North Tower on a maintenance mission. They had called their co- workers to tell them they were escaping to the roof after they realized the extent of the fire caused by the first plane crash.

In the face of such horror, it is difficult to be heartened by the survival of a communications network. But some communications experts said that the ability of networks to adapt in such emergencies could be crucial to the city's recovery.

Consider, for example, what happened that Tuesday in the AT&T operations center in Bedminster, N.J. That day, AT&T handled 431 million voice calls 20 percent more than usual and the most it had ever carried on a business day. The morning of the attacks, blue lines pointing toward New York City lighted up, indicating what the industry calls a "focused overload." Long- distance calls cascading into New York's telephone system threatened local callers' ability to reach emergency services.

The solution was to block some of the calls coming into the city. In industry terms, the strategy is called "call gapping," and it was employed with a few keystrokes by network managers at the control center. Managers typed commands that automatically prompted the AT&T system to keep a percentage of its circuits open for outbound long-distance calls. For people around the country who heard the message "All circuits are busy," call gapping was a likely cause.

"People get emotional about this topic because they think we are intentionally holding back their calls," said Mr. Johnson at AT&T. "The local network in New York City is only going to handle so many calls."

Locally, Verizon was doing its part to make sure that emergency calls could get through. The company provided nearly 2,400 spare circuits to city government agencies (including the police and fire departments), about 900 circuits to the state government and more than 2,600 to federal agencies and the military.

The Internet also served its purpose as an efficient and relatively stable network. Unlike traditional phone calls, which require an open circuit between two people, data sent over the Internet travels in discrete packets that move over hundreds of different channels at the same time and rejoin at their destination, in a process known as packet switching. For the most part, that diverse distribution of the packets allows them to avoid bottlenecks that result from blockages or damage to specific locations.

Many people, for example, had more luck sending e-mail and instant messages than reaching people over the phone because the messages were automatically stored when barriers were encountered and then forwarded to open routes.

Paul Baran, a former computer scientist at the RAND Corporation, who is widely considered the co-inventor of packet switching, said that the Internet's ability to stay up after the attacks was proof that the "route around the trouble" model worked.

As planned, the telecommunications system also relied heavily on built-in redundancies. Many companies, for example, have more than one line from their offices to high-speed access points. But the disaster did expose some of the limits of those contingency plans. Some of those multiple lines travel the same conduits to the same routing centers. If something happens to those conduits or routing centers as did in many cases on Tuesday all the redundancy in the world doesn't help: all the cables would be damaged.

Roy A. Maxion, director of the dependable-systems laboratory at Carnegie Mellon University in Pittsburgh, has long preached the value of physical diversity in networks. "I wouldn't want to be alarmist about this," he said, "but what I think is interesting is how the system is not set up. A lot of these contingency plans are not in place." He added that "as a nation we are dangerously vulnerable."

Right now, telecommunications companies are focused on the task at hand: getting power to buildings, clearing sites of debris, keeping lines up for emergency crews, monitoring the networks that serve the stock exchanges and running new cables for relocated businesses that had offices in the World Trade Center.

"I've been trying to find a way to say it," Mr. Johnson said. "But there is not a soft way to say that the businesses that were normally served are no longer there."

-- Andre Weltman (aweltman@state.pa.us), September 26, 2001

Moderation questions? read the FAQ