Irish Telecom crash due to 'bizarre' software failures

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Irish Telecom (Eircom)experienced a massive crash on Friday due to what they claimed was a 'bizarre coincidence of software failures' (and, no, no not Y2K). Expect more 'bizarre coincidences', coming your way shortly ...

THREE EVENTS COINCIDED CAUSE PHONE CRASH (Irish Times):

Three separate events combined to create the telecommunications crash in Dublin. The probability of all three occurring almost simultaneously would be rated by risk statisticians as extremely remote.

But they did coincide last Friday, and Eircom is left wrestling with claims that the blackout exposed serious shortcomings in its technology and grave call capacity problems - while the spectre of the Y2K computer bug looms in the minds of consumers and edgy business users who fear this is a timely warning of worse to come. Unprecedented, not Y2Krelated, "completely outside the norm", and "there will never be a problem of the scale of Friday again" typified Eircom's reaction.

Glitch 1: On Thursday the phone company made a number of modifications to its Priory exchange in Stillorgan, Co Dublin, which processes land-line calls in what is known as its fixed network. By Friday morning certain categories of calls through the exchange (mainly free phone and premium calls) were not operating properly as a computer software problem manifested itself. Calls continued to build up until the system crashed.

Glitch 2: By Friday afternoon, a second software error arose in a back-up system which prevented the switch-over that would normally occur to relieve the first "processor". Both are Ericsson's "AXE exchanges", which are widely used throughout the US and Europe, and are considered among the most technologically advanced. Eircom is one of the few telecommunications companies in Europe to have fully digitalised networks, which are more efficient and allow for greater call capacity than analogue systems. During the afternoon, Eircom attempted to restart the first system, but the initial glitch prevented this.

A more extensive restart was attempted as calls were rerouted into its Merrion exchange, which in turn crashed. Calls continued to build up and were rerouted to its Crown Alley exchange but it too came under strain because of the domino effect. Eircom later succeeded in overcoming the initial software problem by manually restarting the network at the Priory exchange. This resolved the initial problem and eased the pressure elsewhere.

Glitch 3: In a separate incident, the Eircell mobile phone network (a different telecommunications system) went down due to flooding of its Hely House exchange in the city centre. An underfloor pipe burst so the network had to be powered down as a safety precaution. It was not a capacity problem as such, the company insisted, though mobile calls greatly increased as land-lines were down in much of Dublin. By mid-afternoon calls jammed across the three main Dublin fixed exchanges and the Eircell network was gone. The fixed network was restored by 6.00 p.m. while 90 per cent of the Eircell system was back in operation up to 2 a.m. and fully restored later on Saturday.

The aftermath: Eircom is to carry out a full audit with Ericsson of what went wrong, according to Mr Gerry O'Sullivan, its director of corporate affairs. It did not, however, indicate a capacity problem, he said. It was understandable people would draw comparisons with the Y2K problem, but Eircom "are no less confident today than we were last week" that it was "exceptionally prepared" to counter the bug. It had spent #32 million to protect its systems.

-- Risteard MacThomias (uachtaran@ireland.com), November 22, 1999

Answers

The problems were due to being: Simultaneous ... interconnected ... under conditions never having occurred before. But Y2K will not be any big deal.

Things that make you go Hmmmmmmm.

-- King of Spain (madrid@aol.cum), November 22, 1999.

Moderation questions? read the FAQ