MCI WorldCom Network Still Troubled After 8 Daysgreenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread |
MCI WorldCom Network Still Troubled After 8 DaysUpdated 2:21 PM ET August 13, 1999
NEW YORK (Reuters) - Long-distance phone company MCI WorldCom Inc.'s high-speed data network continued to suffer problems Friday, idling the Chicago Board of Trade's (CBOT) electronic trading system and disrupting service to thousands of businesses for more than a week.
MCI WorldCom's network problem began eight days ago due to a glitch in software provided by Lucent Technologies Inc.
The outages affected about 15 percent of MCI WorldCom's network and an estimated 70,000 customers such as the CBOT, Internet service providers, banks and other businesses that rely on high-speed frame relay networks to exchange large amounts of computer information in short and frequent bursts.
The CBOT, the largest U.S. derivatives exchange, said Friday its electronic trading system, Project A, would remain shut until Sunday night when the exchange moves to its backup system indefinitely.
"Market users worldwide depend on Project A around the clock and MCI WorldCom has let them down for one full week. As a result of MCI WorldCom's failure to deliver on their promises to me early last week, the CBOT is pursuing all available remedies," CBOT President Tom Donovan said in a statement.
Another customer, a major U.S. retailer, said MCI WorldCom's network service began to come back during the middle of this week only to grind to a halt again Friday morning.
"Things started to get better, but there was some failure overnight and now we are in worse shape than ever," said one MCI WorldCom customer who declined to be named.
The customer said the network problems prevented him from communicating with distributors, other company locations and his clients.
"I've never seen a network outage last a week. If they had been more clear in the beginning about the extent of the problem or their ability to handle it, we could have made other contingency plans. But now we're just going with day-by-day information and that's not helping," the customer said.
Clinton, Miss.-based MCI WorldCom said the network is now stable, but problems arose last night during some network maintenance. That instability may have made service problems worse for some customers, but service quality should improve today, said MCI WorldCom spokeswoman Linda Laughlin.
The company said it continues to restore service to customers and to talk to clients one-on-one about the problems and any potential compensation.
MCI WorldCom's outage followed a similar problem at AT&T Corp. last year. AT&T and other companies such as Qwest Communications International Inc. said they have gained customers this week as some disgruntled MCI WorldCom clients searched for new service providers.
============================================= End
Ray
-- Ray (ray@totacc.com), August 13, 1999
Yep.... I just managed to log on again to Compuserve after 3 days of outage. The most frustrating part was not even being able to get through to CS - kept getting recording that I would have to "try your call again later". They were swamped. The first 2 days I was told all sorts of incorrect stories but finally yesterday was told it was due to WorldCom's failure. I also was disgusted that I couldn't be told an estimated repair time. Thankfully it is back up. Supposedly credits for the days lost are supposed to be forthcoming.... we shall see. Never saw any mention of it on CNN or Fox news..... hmmmmm - not a big enough story, ehhhhhhh?
-- Kristi (securxsys@cs.com), August 13, 1999.
Thank goodness this cannot possibly be Y2K-related, because fix on failure due to Y2K would only take a few hours...right?
-- RUOK (RUOK@yesiam.com), August 13, 1999.
"If they had been more clear in the beginning about the extent of the problem or their ability to handle it, we could have made other contingency plans."That sounds familiar.
Got contingency plans?
-- Deborah (infowars@yahoo.com), August 13, 1999.
Is the house of cards crashing? Marie where are you???
-- y2k dave (xsdaa111@hotmail.com), August 13, 1999.
Maria's over at diBunki today, safe in the knowledge that her polly pals will never broach the topic.
-- lisa (lisa@work.now), August 13, 1999.
This one should be getting shitloads of attention. Thus far only a brief mention on Talk Radio. There are thousands of companies worldwide being directly affected by this one.I have firsthand experience with it. My friend at "big consulting overcharger for everything llp." told me that they have clients all accross America and Globally who are screwed good by this.
Why oh why is it not front page news? And the stock went up. Duh.
-- Gordon (g_gecko_69@hotmail.com), August 13, 1999.
Old thread where MCI WorldCom has August as hopeful remediation installation month. Interesting!
-- lisa (lisa@work.now), August 13, 1999.
Yup, we were too. Unable to do business yesterday that is, "all circuits are busy", all day long.
-- Dependent on it (allcircuits@busy.com?), August 13, 1999.
Just a quick rib --Where are all those contingency plans? Where are the manual backups? Let's just run it manually!
-- Larry (cobol.programmer@usa.net), August 13, 1999.
This problem is related to the eclipse. Plus the customers were not wearing their foil hats. I kept telling them but they wouldn't listen.
-- Maria (anon@ymous.com), August 13, 1999.
Lisa,Nice catch! Here are some memorable quotes from the thread that you referenced (I particularly like the last one!):
"...MCI Worldcom projects a Sept. 30 date for "deployment of remediation solutions" (that's cutting it close, and it's not clear if that includes testing)..."
-- Drew Parkhill/CBN News (y2k@cbn.org), February 23, 1999
"Consider MCI TOAST! Put a fork in her she's done!"
-- Scotty (BLehman202@aol.com), February 24, 1999.
"MCI and Sprint don't really need to TEST those millions of lines of remediated code, do they?..."
-- Nabi Davidson (nabi7@yahoo.com), February 24, 1999.
"Hmm, you guys need to know that its not necessary to test the H/W & S/W remediations on a "live" system. All telecomm infrastruc- ture providers employ Test Plants (that is mirror images of their customer's networks) that allows them to not only test corrections for problems like y2k but new features before releasing them..."
-- kevin (kta@yahoo.com), February 25, 1999
-- RUOK (RUOK@yesiam.com), August 13, 1999.
and the disinformation wheel keeps turning round and round...
-- Maria (anon@ymous.com), August 13, 1999.
... while the receiver keeps emitting "all circuits busy, please try your call again..."
-- lisa (lisa@work.now), August 13, 1999.
Interesting commentary about how long it takes for customer defections. 10 - 20% customer defections in one week. ISPs discussing shutting their doors. In the Internet age, switching costs and time are reduced, but will there be an alternative to switch to?http://www.news.com/News/Item/0,4,0-40499,00.html?st.ne.lh..ni
ISPs say MCI outage could kill businesses By John Borland Staff Writer, CNET News.com August 13, 1999, 2:00 p.m. PT Steven Efurd is afraid of losing his seven-year-old Internet service provider business in Tyler, Texas.
He's lost his connection to the Net for eight days now--the first extended outage in his company's history, he says--and hundreds of his customers are beginning to flee.
And there's nothing he can do about it. Like dozens of other small ISPs around the country, Efurd's East Texas Global Services relies on MCI WorldCom's data network to connect to the Internet. That company is still suffering a crippling nationwide network outage, putting Efurd out of luck--and, if it continues much longer, out of business.
"If I go through the weekend without Net access...I might as well lock the door and go home," he said.
Around the country, scores of small ISPs that depend on MCI WorldCom's data network to connect to the Internet are offline. Like Efurd, many are bleeding customers at rates they say will shortly kill their businesses.
In Plainview, Iowa, Powerline Internet says it has lost close to 20 percent of its 600 customers in the last week. A.K. Jordan, president of Intellitech in Highland, New Jersey, says he's already lost about 10 percent of his 1,000 customers but can't really tell how badly the company has been shaken yet.
"I can't even gauge how much bleeding there is, because email is down," Jordan said. "Customers can't even send me a nasty email about the problem."
The problem has been compounded by the way many ISPs buy their connections from wholesalers. RMI.Net, a Colorado-based service provider, sells data connections to 150 ISPs nationwide. Because its backbone connections have been caught in MCI WorldCom's problems, 66 of its customers were still down as of midday today, RMI spokesman Mark Stutz said.
Those customers in turn serve about 200,000 to 250,000 subscribers, Stutz added. And like many other companies, RMI says MCI WorldCom has not been forthcoming about the problems or helpful in trying to mitigate their effects.
"MCI WorldCom has the Internet's version of the Exxon Valdez on its hands, and they're acting like a few turtles got a little oil on them," Stutz said. "This is about as bad as it could be."
A week of network nightmares MCI WorldCom's woes began more than a week ago, when a planned software upgrade to their high-speed data network went awry Thursday evening.
The problems are grounded in the frame relay network--a data technology similar to the public Internet that breaks files and messages into individual "packets" of information and sends them separately to their destination. These networks are heavily used by businesses, banks, and ATM machines.
The source of the problem was a software upgrade to the Lucent Technologies hardware that routes data though the networks. Teams of engineers from Lucent, MCI WorldCom, and Bell Labs have been working on finding the source of the problem all week but still haven't isolated the root cause, a Lucent spokeswoman said.
Since midweek, the company has been saying it has been contacting customers individually and bringing them back online one by one. A spokeswoman said the original outage affected 15 percent of the company's network and 30 percent of its customers, but that number is dropping as fixes are made, she added.
But many customers, ranging from small businesses to the Chicago Board of Trade, have complained bitterly that MCI WorldCom's efforts have been inadequate.
That dissatisfaction, which is spreading by the day as the outages continue, has raised the specter of wide defections from the company's data business. In 1998, AT&T had a similar outage which lasted for several days, but CEO C. Michael Armstrong quickly took public responsibility for the problems. That helped keep all but about 2 percent of its customers on board, analysts said.
But while MCI WorldCom tries to fix its problems, ISPs say their customers are blaming them--and until MCI WorldCom makes a more public statement about the scope of its problems, there's little the small companies can do.
"Because MCI is not forthcoming in the press, when I say it's not us, it's hard to get someone to believe you," Jordan said. "They're saying its congestion in their network. This isn't congestion. This is not a little problem. It's shutting my door."
Some ISPs are talking about lawsuits but are reviewing their contracts and waiting until the problem is fixed to see what MCI WorldCom ultimately does.
MCI WorldCom says it doesn't want to discuss the problems until they know what caused them. "Until we find out what was the root cause, we're not going to guess what caused it," spokeswoman Linda Laughlin said.
But every day the telephone company maintains its public near-silence hurts the ISPs, they say.
"My reputation has gone to hell," Efurd said. "I try to be honest with everybody, but of course they think you're lying. And how do you recover from word of mouth?"
-- Net down ISPs lose (txprepper@yahoo.com), August 13, 1999.