Actual Y2K Failed Tests?

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

For the last six months I've been reading lots of editorials and speculation about Y2K problems in the electrical industry. What I haven't seen are actual reports of Y2K related failures during the tests which, presumably, many of the utilities are conducting. what can you tell me about it?

-- Anonymous, August 06, 1998

Answers

The following article may offer some answers to your question. R.A. Mann

Power grid in grip of testing paralysis By JOHN MACLEAY The Australian 06aug98

WHETHER the electricity remains on after January 1, 2000, is the big unknown of the millennium computer glitch, says power industry consultant John Catterall.

The West Australian manager for Infrastructure Control Systems, Mr Catterall says if the computers controlling the electricity grid fail, the consequences could be nothing short of catastrophic.

Yet, to date, the electricity industry has been reluctant to test for the problem.

"It's the most important aspect of year 2000 compliance. If the electricity doesn't make it, and we have significant problems in the electricity industry, all the work being done elsewhere will have been a waste of time," he says.

Mr Catterall, who was previously in charge of the compliance program for electricity utility Western Power in Perth, says if the grid goes down for more than four or five hours because of the bug, then the power could, potentially, be off for at least several weeks until the problem is fixed.

At the centre of Mr Catterall's concern is the system controlling the grid, the SCADA, short for Supervisory Control and Data Acquisition.

"Essentially, if you lose control of your SCADA system, you lose control of the grid because you won't know what's going on on it," Mr Catterall says.

A SCADA system, in essence, gathers information from along the grid to determine how it is performing and how output can be adjusted to match load demand to ensure maximum performance. And, like the rest of the power industry, it contains many date-dependent computer chips.

The interconnectedness of the system means that a fault  just a few faults  could be enough to bring down the SCADA and, with it, the grid.

In the event of a computer failure, a grid without a SCADA would be "flying blind" and vulnerable to damage unless closed down.

"They are extremely vulnerable to the year 2000 and, depending on how the information on the grid gets back through the SCADA, the potential could be catastrophic," Mr Catterall warns.

He says some utilities do not appear to be adequately addressing the year 2000 issue. They have still not fully appreciated the scope of the problem, judging by their faith in the guarantees of millennium compliance given by the vendors of both the software and the computer hardware that control the grid and power production.

He says there has been a reluctance for utilities to wind forward to 2000 to see what could happen.

That reluctance could be due to the fact that any millennium change problems may come early, as happened in New Zealand with its grid authority Transpower.

Transpower is still determining what went wrong even though outside consultants had given that grid system the all-clear as being year 2000 compliant before it was tested.

Mr Catterall says the major benefit of testing now rather than waiting until January 2000 is that, in a controlled and tested environment, problems affecting the grid can be fixed far more readily.

If a component is needed, then there is more chance it can be ordered and installed quickly.

After January 2000, the rest of the economy will be caught up in dealing with its own bug problems, making repairs and getting technicians to the site of grid breakdowns all that much harder to achieve.

Mr Catterall's experience at Western Power was that once a company began to appreciate the extent of their year 2000 problems, they then began to realise they could not guarantee full compliance but only attempt to mitigate the worst effects of the date changeover.

Mr Catterall says the power industry needs to thoroughly test its SCADA systems but there has been a commercial and political reluctance to do so.

Although a notional national grid now exists between NSW, Victoria and South Australia, each State would have to test its own grid.

However, a meeting of electricity industry executives in Melbourne later this month is expected to outline a timetable for tentative national testing arrangements.

NSW Transgrid chief executive Warwick Grainger says his organisation  which controls that State's grid network  expects to have all its critical systems year 2000 compliant by the end of 1998.

But Mr Catterall warns that time is running out. With summer and winter as peak load times, the only period of the year with adequate excess capacity to allow power stations to shut down, and back-up grid control systems to operate, are spring and autumn.

"Time is running out and there still is no co-ordinated plan to get the generation aspects of year 2000 certified," he says.

Mr Catterall points out that the grid is not the only source of his year 2000 worries.

Power stations are "chock-a-block" full of embedded systems, that is, silicon chips that control electricity production that could be affected by the millennium bug.

He says he knows of only one power station that has been tested for millennium bug compliance by winding the date forward to year 2000. That was in Britain and the power station failed.

) News Limited 1998

-- Anonymous, August 06, 1998


So, we have lots of rumors and "urban legends", but no documented cases of power plants actually failing during testing. I still have not seen any documented failures of power plants or grids directly relating to Y2K testing. I can see the huge potential for catastrophe here, as well as the need to inform people or the problems. However, I think "rumors" and "urban myths and legends" will lessen the impact of the warning.

So, again, is there any documented cases of Y2K testing failures? How about the one in Hawaii? Did that happen? What happened in Britain? Some documentation will go a long way in raising awareness.

-- Anonymous, August 24, 1998


I would also like to see documentation on any failure that presumably happened. So far the media and high profile politician, ceo's , etc. have repeated what they have been told to say without provide substance. This has fueled panic and uncertanty which can only suggest that the media should talk to engineers, computer scientist, computer engineers and computer system designers.

-- Anonymous, August 24, 1998

Most of the electric utilities are testing mission critical systems. The tests are conducted after applying the software patches. Hence failure during a Y2k test will be minimal. Take the case of Global Positioning Satellite (GPS) Week Rollover problem which happened on Aug 21/22, 1998. None of us ever experienced a grid black-out, ships colliding, airplanes landing elsewhere and other gloom and doom episodes. As days roll by, Mission Critical systems will be tested and there will be minimal problems. We can expect some nuisance problems, though.

-- Anonymous, August 24, 1998

Mr Rajsekar, you're showing your lack of knowledge of the issue - GPS roll over doesn't happen until August, 1999.

-- Anonymous, August 24, 1998


Check my site for some documentation from GPS synchronous clock manufacturers http://home1.gte.net/esteban/GPSBUG.htm Esteban <\\\><

-- Anonymous, August 24, 1998

The date mismatch was a lapse on my part. The point I was trying to drive was that Y2k testing will throw plenty of minor or nuisance problems. Examples like: a Controller unit which doesn't roll over correctly will still continue to function normally, but an event log will print a wrong date. If this controller is a mission critical grid Disturbance Recorder, a faulty event date will not serve any purpose. In such cases a software patch has to be applied before testing. Take another case of a Grid Load Forecasting software -- if the date rolls over to 1980, the forecast will be erroneous and the data will be rejected by a Energy Management System. A sofware patch should be applied before commencing the tests. The same concept applies to GPS -- if the receiver is used for only geographical coordinates (like Ship navigation, airplane radars, etc.), an improper date can be a nuisance. On the other hand if the GPS receiver is used for absolute time stamping (like SCADA/ EMS, DCS, Telecom, etc.), an improper date can result in loss of data synchronization among different modules. The severity of the problem depends on the configuration of the SCADA/ EMS or DCS. The North American Electric Reliability Council (NERC) is shortly planning to publish at its website, the inventory and test results of mission critical items from various electric utilites. The test results can give good confidence to common public.

-- Anonymous, August 25, 1998

Mr. Rajsekar:

I glady submit that you are more knowledgable than I in this arena. Having said that, what evidence do you have to support the concept that an embedded chip, once reaching the millenial turnover, will reset to 1980? Has it been my mistaken impression that with micro-processors, they would reset to "00" instead of "1980" and consequently risk shutting off? Thank you.

-- Anonymous, August 25, 1998


Many chips roll to an April, 80 date... a common epoch date for chips.

-- Anonymous, August 30, 1998

Moderation questions? read the FAQ