Y2K Test KO's Phone Service http://www.ottawacitizen.com/business/990723/2640543.html

TORONTO - A Y2K glitch yesterday caused yet another Bell Canada system to crash, leaving thousands of Ontario businesses and residences without phone service ...

Mr. Lalonde said 7,500 Toronto PrimeLine customers were being transferred to a new Y2K-ready system. It operated for only three hours before crashing at about 10 a.m. local time. The problem lasted two hours.

About 1,200 PrimeLine customers in Ottawa-Hull, Hamilton, London and Kitchener who were already on the Y2K-ready system for several weeks, were also affected ...

It was the third time in a week that Bell experienced a system failure, with the collapse of a Toronto-area 911 emerbency network on the weekend and a fire last Friday that affected service nationwide ...

-----------------------------------------------------------

PHONES GO DEAD, TORONTO PUT ON HOLD Fire in Bell Canada switching station knocks out more than 100,000 lines Saturday, July 17, 1999

EXCELLENT article detailing cascading effects throughtout Canada.

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=0016F2 -----------------------------------------------------------

GET ROTARY PHONE - HERE'S WHY ...

The ripples were pure idiosyncracy. They missed much of Montreal, but hit Halifax, Vancouver, Chicago. They knocked out touchtone phones but not rotary dial phones - for those who remember what they are.

Along with the billion, we took an even bigger hit in our confidence about Y2K. And even before our vulnerability could fully flower, we got hit again. A Bell equipment ``glitch'' in Peel left a million people without 911 service for much of Sunday.

And - oh, devious technology - many of those who dialled 911 without result found they couldn't call anyone else for help, either. That's because the 911 system is designed to maintain connections to trace calls, even if callers hang up ...

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=0017Gg

-- Cheryl (Transplant@Oregon.com), July 23, 1999

Answers

Great post Cheryl. Just happen to have an old rotary phone i can get my hands on for free......JIC

-- kevin (innxxs@yahoo.com), July 23, 1999.

Cheryl, wouldn't you rather see these things happen now? While they are inconvenient, this is what testing is all about. Some rather important stuff you englected to post...

Bell says it expects no further Y2K problems.

"To date we have been successful in upgrading 99.9 per cent of our network with no impact on customers," said Mr. Lalonde, adding that represents 325 central offices. He said Bell expects to be Y2K-ready by early fall.

-- Y2K Pro (y2kpro1@hotmail.com), July 23, 1999.

Of course we've all expected and are seeing evidence of Y2K testing. This is the year for it.

These "glitches" indicate the problems are REAL and at do cause additional problems.

One of the concerns that I've heard from a reliable friend is that * some* of the smaller regional phone areas (definately in his) won't receive replacement phone switching equipment until the fall because of ordering backlogs. Some won't have time to install the new Y2K- compliant equipment in time before 12-31-99.

Expect some regional telecommunication problems and prepare accordingly.

BTW, I always keep one of the older "princess-style" phones that don't require an electrical plug-in adapter to keep it operational. During power out times, living in the mountains, I was still able to call in and report outages in our area.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), July 23, 1999.

Quick question for those who know something about "telephony" (what a interesting word that is. . .):

I have an old (by today's standards) trimline phone manufactured by A T & T, that has a touch tone feature and a rotary dial feature.

There's a switch. On one side it says "TT." On the other, "DP."

When the "DP" switch is activated, the dialing process sounds as though it were taking place on a rotary phone.

Is this the same thing as HAVING a rotary phone?

Also, what does that "DP" stand for?

Thanks!

:)

-- FM (vidprof@aol.com), July 23, 1999.

Y2kPro, something YOU neglected to mention. Bell said they expected NO problems BEFORE this happened.I'm sure this was just an over-sight on your part, not deliberate, LOL.

-- River (riverwn@aol.com), July 23, 1999.

Hi Maria - just a reminder. When Bell loudly announced they were compliant (a few weeks ago!) I pointe out that they (like the FAA, but the major phone systems are even further advanced in remediation) had only completed the first two of the four steps: lab back-to-back testing of the main switch and routing equipment, and company-company simulated routing of the main programs and equipment on a test network machine.

Phase 3. They still needed to actually install the new programs and equipment, and then (Phase 4) complete integrated systems testing on the "real" components and "real" programs - using a test program to simulate real messages, or in parallel with regular phone service.

Most important. Regardless of the y2k remediation and testing phases, eventually you must actually put the new system(s) in service. (This is what the FAA has been trying to do without the testing part.)

_____

Ultimate y2k systems requirements.

One. Complete remediation and all testing. And maintain current levels of service in 1999.

[Appears the phone system, and FAA, can't quite get this step done yet!)

Two. Be able to provide equivilent levels of service year such that the company can remain in business next year, preferably maintaining or increasing its profits.

[To answer Paul Davis challenge in other places, we'll know this by end of first quarter of 2000. We'll know how effective the federal remediation effort has been by 15 April 2000, but the proof will come on election day, Nov 2000.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), July 23, 1999.

FM - DP means Dial Pulse yes it is the same as a rotary phone. the physical rotary phones had and have a mechanical spring that returns the dial to starting point, along the way, it makes and breaks contacts, the new digital phones operate a relay that opens and closes the contacts the same way.

Phone systems can not tell the difference between the original rotary dial phone and the new phones in DP mode.

The Switch was there for two reasons.

1. It took some time before all markets were touch tone available

2. when you still had the old dial pulse service, once you dialed your destination number, many services were available on the other end with a touch tone menu, so you dialed pulse, then switched to TT to access your bank account etc. vnqbdtnece w

-- Living in (the@real.world), July 23, 1999.

I can give you another man-on-the-spot report on this one as well.

My business uses Bell's Prime Line as well, and one of my staff reported problems with it about 11 am. It didn't cause any disruption to us, as we only use it to allow some of our customers in outlying areas of Toronto to call our stores as a local call. They can always revert back to our regular toll line or our 888 number. I suspect that the majority of people who use Prime Line use it in the same way we do.

As to the bigger Y2K picture, I'm with Y2K Pro in that I'd rather see this kind of stuff now than in Jan 2000. If there has been a heck of testing and implementation going in with Canadian telecoms then there have been very few incidents like this. What I don't know - cause I'm not on the inside - is how much implementation is going on. I suppose that is the C$64,000 question.

One final note on Prime Line: about 4 months ago we got a letter from Bell saying that they were going to be cancelling the Prime Line service at the end of Dec 99 (they said something about how it wasn't as popular as they had thought it would be). I was annoyed as it had proven to be very useful to my company. I didn't complain in writing, but enough customers must have complained because about 2 months ago I got a letter saying that Bell was now *not* going to cancel the service.

So, my hunch is that Prime Line wasn't Y2K compliant and there were no plans to remediate it as it was scheduled for cancellation before 1/1/2000. However, once an outcry forced Bell to keep Prime Line beyond 1999 they had to make it Y2K compliant. Thus the little glitch just reported.

I think it would be wrong to draw inferences about Bell's overall readiness from the Prime Line incident.

-- Johnny Canuck (j_canuck@hotmail.com), July 23, 1999.

Thank you kindly, Living in.

I remember now, that I bought that phone for my mother back in the 1980's, before she had touch tone service.

:)

-- FM (vidprof@aol.com), July 23, 1999.

Well, Johnny, I've got GREAT news for you and Y2K Pro: You will have it BOTH ways, my friends. You will see these incidents happen NOW as companies fruitlessly attempt to test their "remediated" systems, and you will ALSO see these incidents go supernova in 2000 as everything falls apart! (Of course, there is always the "Van Nuys" approach: If a Y2K test makes a big, awful mess that you can't hide, simply SUSPEND ANY FURTHER TESTING!!!)

-- King of Spain (madrid@aol.com), July 23, 1999.

Cheryl, wouldn't you rather see these things happen now? While they are inconvenient, this is what testing is all about.

Huh?

They weren't testing, they were implementing.

-- Lane Core Jr. (elcore@sgi.net), July 23, 1999.

Your Majesty

Upon what bases are you making your assertions:

"companies fruitlessly attempt to test their "remediated" systems ... go supernova in 2000 as everything falls apart" ?

On the basis of extrapolating from a small number of publicised Y2K testing/implementation failures so far in 1999?

On the basis that if very little is being said publicly about remediation and testing then there must be very little remediation and testing actually happening?

Rather broad, sweeping statements like yours don't really add much to the discussion. [This doesn't mean that you don't have a right to post such opinions - I'm not in favour of forum censorship.]

Johnny - who is bowing and walking slowly backwards out of the room.

-- Johnny Canuck (j_canuck@hotmail.com), July 23, 1999.

Question:

"They knocked out touchtone phones but not rotary dial phones - for those who remember what they are."

What about people who had TOUCHTONE SERVICE (which is probably most of the people) but ALSO have a rotary phone? It seems that if the problem was with Bell, and not with the phone, then even people who have a rotary phone IN ADDITION to any touchtone phones they have, would lose service. Yes? No? Explanation?

-- Linda (lwmb@psln.com), July 23, 1999.

Touchtone is an option that requires additional equipment at the telco office. Roatay phones require only basic service, and work on any line. At least I think so! <:)=

-- Sysman (y2kboard@yahoo.com), July 23, 1999.

Pro -

Of course we'd rather see these events now, while resources and time remain. The fact that they occurred during deployment (as in "production installation"), despite Bell Canada's best efforts, leads one to conclude that other, similar events will occur despite the best efforts of the project teams. That's why testing should be (but too often is not) the biggest phase in the project plan.

That was just a slip of the keyboard, wasn't it? Given your monicker, you could not really have meant to characterize "implementation" as "testing". Failures during implemenation/deployment cost a factor of 10 more than failures caught during final integration testing.

Or did you think that Bell Canada's customers were doing "acceptance testing" and just didn't get the memo?

-- Mac (sneak@lurk.hid), July 23, 1999.

Mac:

The real world is messier than you imply. All too often, the line between testing and implementation is very blurry. In a very real sense, every implemented system everywhere is 'under test', that's a lot of what maintenance programming is all about (unfortunately).

So the real world strikes again here. The bad news is that an implementation had problems. The good news is that (apparently) only one of 325 implementations had problems and only for two hours. The bad news is there are a *lot* more implementations left to do.

And so it goes.

-- Flint (flintc@mindspring.com), July 23, 1999.

Just a few thoughts on the Bell fire last week. The reports say it was caused by "a workman dropping a tool".

If this is true, wouldn't he notice that he'd caused a fire? Wouldn't you expect him to be able to extinguish it before it got out of hand? How much flamable material is in the near vacinity of most electronic equipment? Seems to me its usually a mass of wires etc.

I can't help but speculate that perhaps the "workman" was actually a Y2K programmer running tests in the middle of the night when traffic was low. He was quite possibly on a different floor of the building, in the computer center.......could the tests have caused a failure in say, a cooling fan, which would allow overheating in a switch or something.....resulting in a fire. Very possibly, the "workman" wouldn't even know about the fire until it had done some serious damage......AND caused the fire alarm/sprinkler system to activate.

It's just STRANGE to me that Bell suffered this fire on Fri, a loss of 911 system on Sun, and then admittedly Y2K implementation problems on another system the next Fri. THREE major problems in one week. Hmmmmmm.

Maybe I'm just being too suspicious.......but I would expect a "dropped tool" to perhaps short something out, blow a fuse, or physically damage some electronics. I just find it hard to believe that a "dropped tool" could cause a major fire unless there was something highly flamable in the vacinity. Besides, its amazing to me that a "repairman" was in the bldg. at 6:00am. Unless its an emergency situation, seems to me most repairs are done during working hours.

On the other hand, I WOULD expect Y2K testing/implementation to be done during the middle of the night or on weekends.

-- Sheila (sross@bconnex.net), July 23, 1999.

Sheila, I agree. We can't prove it, can we? I still agree...

Johnny, common sense is the key...

-- George (jvilches@sminter.com.ar), July 23, 1999.

Maybe this is Y2K related too ....

http://www.tcpalm.com/portstlucie/v15sphon.shtml Thousands disconnected July 15, 1999 By Paul T. Rosynsky of the News staff

PORT ST. LUCIE, FL - What was intended to give BellSouth customers improved service gave them headaches instead Wednesday as an ''upgrade'' at a regional transfer station disrupted telephone service to 187,500 Treasure Coast residents.

The outage, which affected everything from St. Lucie County's 911 system to take-out pizza orders, began about 1 a.m. and spread from Port St. Lucie to Sebastian.

Although service resumed for most customers by 7 a.m., some areas of Port St. Lucie weren't fully restored until early afternoon, BellSouth reported.

The problem began in the transfer station, a concrete bunker-like building off Port St. Lucie Boulevard just west of U.S. 1, where three technicians were working to give BellSouth computers more power. The intent was to make the computers better able to handle sophisticated operations such as call waiting and three-way calling.

When the technicians connected new electrical lines to a bank of high- powered batteries, the resulting power surge appeared about to exceed acceptable limits - which could have destroyed the entire computer system.

So workers immediately shut down the system, cutting service to 187,500 customers whose phone lines go through the regional station.

''What caused the haywire was shutting it off,'' said BellSouth spokesman Rob Seitz.

Added regional manager Sid Poe, ''It's kind of like when you hit a brick wall. When it went down, everything stopped.'' In Port St. Lucie, lines for 17,500 customers - exchanges 337, 335 and 398 - originate at the station, and those customers lost dial tone and all service. An additional 170,000 customers in Fort Pierce, Vero Beach and Sebastian lost regional and long-distance service routed through the station but could make local calls.

The shutdown sparked a series of frantic phone calls to BellSouth officials from West Palm Beach to Atlanta to determine what happened and find a solution to what Poe called an ''unusual'' outage.

The hum of computers and the high-pitch noise of running batteries was transformed into eerie silence as technicians feverishly worked to restore power. More than a dozen technicians converged on the station in the early morning to bring the computers back online.

But unlike a personal computer, which reboots itself in seconds, technicians had to carefully inspect every piece of equipment, a process that took hours, spokesman Seitz said. ''Unfortunately, its not like an on-and-off switch,'' he said.

-- (Too Shy@ToSay.com), July 23, 1999.

Yes, Too Shy, it sure smacks of Y2K, but does not say. And the reality is that NOBODY KNOWS just how much Y2K "testing" is responsible for problems such as these.

One thing that KNOWN tests that go awry show is this: Y2K is NOT a trivial problem, it is NOT something that -- if not fixed -- will be a simple "bump in the road". Especially when these "bumps" happen all over the place at the same time.

Johnny, you asked on what basis I was making my assertion regarding the status of Y2K remediation and testing. I will give you two words: common sense. Its late July 1999, dude. Wake up and smell the coffee. (Flint is hopeless, no sense in wasting good java.)

-- King of Spain (madrid@aol.com), July 23, 1999.

What I heard from a friend in Toronto is that the first outage happened on Thursday. A diesel generator was fired up to provide power. On Friday the generator caught fire and blew up and things went down. A police chief used a pay phone to give status to radio stations. The banks shut down and a lot of people went home early.

-- John Littmann (JTL9700@JUNO.COM), July 23, 1999.