Power companies hum along on testing

Thursday, April 1, 1999 STEVEN CHASE Alberta Bureau Calgary -- Jan. 1, 2000, has already come and gone without much fuss for many power plants across Canada, from Nova Scotia to Alberta. Across the country, power-generation facilities are being fooled into thinking it's Dec. 31 and are rolling over their internal clocks -- so far with few problems. For instance, Nova Scotia's five generating stations are already working in the new millennium. In fact, clocks in some of the stations are already halfway through 2000. It is successful testing of power-plant computers like this that has left electric utilities increasingly confident that the bug's bite won't cause major blackouts. "We're very comfortable," said Beverlee Loat, spokeswoman for Edmonton Power Inc., a utility serving the Edmonton area. Most electric utilities are now testing the transmission systems that funnel power to different regions of their market, again with no major problems so far. Most are expecting to finish much of their critical fixing and testing by midyear. Power companies are now running Y2K-susceptible devices as a group. "It's like we've fixed every instrument in the orchestra and we want to make sure we can still make music together," said Bill Imms with Ontario Hydro in Toronto. No power company can guarantee there won't be interruptions on New Year's Eve. Power companies are vulnerable to such things as power surges should large industrial users shut down unexpectedly during the change over, leaving the system with too much power going nowhere. Utilities, including Ontario Hydro, will spend New Year's Eve with emergency response teams ready for unexpected problems. Staff would manually operate systems that are usually automated. Satellite phone links will be available in case land lines fail. A Royal Bank of Canada survey of Canada's power grid concluded nearly all utilities have a low probability of power failures. Separately, gas utilities say they expect no disruption to gas distribution and will have staff at gas compressor stations, which move the fuel along pipes, to manually operate things if necessary.

-- Norm (nwo@hotmail.com), April 01, 1999

Answers

>gas compressor stations, which move the fuel along pipes

When I called my gas utility last year to ask about Y2k compliance, first I was told there was nothing involved in gas distribution that could be affected. When I then pointedly mentioned embedded systems in pumps, the fellow told me there were no pumps involved. He refused to consider the compressors in the pipelines from which his utility received gas into its reservoirs to be part of the picture.

-- No Spam Please (No_Spam_Please@anon_ymous.com), April 01, 1999.

Nothing involved? Same things as every other distribution system - and the same potential failures.

By the way Norm - of the last 25 things you posted - none had any original thoughts, none offered anything but a cut+paste of a reporter's quotes from the national releases, and none provided any new information. No numbers, no comparable facts, no progress - just vague feel-good reassuring phrases from the same sources. And 21 of those 25 repeated articles already referenced.

Just what is your agenda? What do you gain from this effort to demean and distract people from legitimate preparations.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 01, 1999.

Norm's Theme

-- ? (?@?.?), April 01, 1999.

Norm's

Theme

-- Not Norm (happy@face.boy), April 01, 1999.

Thanks for the information Norm (I had not seen this article before). The article is a good general assessment about what is happening with power plants across North America. To date, over 50 plants have had successful Y2K tests, and more are being tested weekly.

Regarding the gas distribution system, it is true that the majority of their equipment is date-immune. However, most gas companies should have some sort of control systems that have date-awareness, so it is inappropriate for a gas company to not have a more detailed response about what work it has performed on Y2K.

Mr. Cook, is there something in the article that you disagree with? Have you been involved in any Y2K plant tests? What are your concerns?

Dan.

-- Dan (dgman19938@aol.com), April 01, 1999.

Norm,

Keep on keeping the info flowing. We need all sides to make the final decisions. Each will make his or her own assessment of the article.

Keep the faith!

BW

-- Bob Walton (waltonb@kdsi.net), April 01, 1999.

yes Dan - I've built 'em, run 'em, tested 'em, rebuilt 'em, and dismantled 'em. I've planned outages, budgetted outages, and re-tested the plants and distribution systems after tests failed.

Many times.

Your background - seen in the earlier thread - is perfectly valid in the small corner you have seen of embedded systems - the bigger picture - where "whole systems" must work - is very different. Here - "Norm" has only one valid point about one utility that has tested adequately (Ontario Hydro). That's it. The rest is a simple repeat of "feel good" statements - and NOT "many plants adequately testing across the whole country."

What is adequate? Look at the thread "What would it take to convince ...." There is a short list of requirements there - either the administration does that minimal level of integrated testing described - or it must expect many irregular, system-wide, unusual and irregularly lasting outages over wide-spread areas of the US and Canada.

Norm (new world order @ hotmail.com) has (again - for the 25th time in seven days) here repeated his somple babbling pattern of mindlessly copying old newspaper reports that do not offer any new information.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 01, 1999.

Thanks for your response, Mr. Cook.

I searched for the "What would it take to convince..." thread, but I couldn't find it. Can you help?

Yes, it is true that power plants are highly complex, but as it relates to Y2k, not nearly as much as you imply. In order for there to be a "system" issue, there must be more than one device that uses date functionality, and passes that date information from one device to another. In general, the transmitters sending information to the DCS at a power plant do not do this.

Regarding my "small corner" of embedded systems, it's not small at all. I have personally Y2K tested over 100 embedded type systems in the transmission and distribution systems, making up the full complement of any large power company, so I have a full understanding of their implications and readiness. I've only personally tested a dozen or so devices at a power plant, but my fellow team members at the plant have reported similar success rates in their testing of about 200 device (and system) types. The 50 or more power plant rollover tests performed by dozens of other utilities suggest that there are very few problems out there once a minimal amount of remediation is performed (typically an upgrade of the existing DCS).

In my opinion, because power plants in general do not care what the date is, they are "innocent until proven guilty", in a sense, not the other way around. Your statement "irregularly lasting outages over wide-spread areas of the US" greatly mis-characterizes the test results we have found so far, and is in my opinion unnecessarily negative. At present it appears that outages will be very minimal.

The NRC will be performing extensive audits soon at several nuclear plants, so perhaps we'll know more about that industry when the reports come out.

My question earlier was not necessarily whether you've worked at power plants in the past, but have you Y2K (repeat, Y2K) tested any power plants? This is not a flame, I'm just trying to understand better where you are coming from.

Dan (I'm not a PE yet, I'm just a lowly EIT)

-- Dan (dgman19938@aol.com), April 01, 1999.

Dan: What would it take for you to change your mind? <:)=

-- Sysman (y2kboard@yahoo.com), April 01, 1999.

Those in overseas systems (Korean, Japanese) - but what I meant by "small corner of embedded systems" is NOT to count numbers - EPRI confirms only about 2-3% are affected - you've reviewed about that many - and found none (good!) - which is statistically in the right order of magnitude.

The effect from Y2K isn't in one part, one chip, or even one power plant - it is in the whole system which is depending on many hundred systems operating successfully in many thousand places all at the same time time. Failure doesn't mean catastrophe everywhere - it means - like Yourdon pointed many thousands mosquito bites, a few hundred small cuts, a few big things that must be stopped eventually, and maybe one or two that can kill you if the bleeding isn't immediately prevented.

The failures will be due to widespread simultaneous failures you don't have experience in - from factors you aren't aware of yet - because they have not been tested yet.

Remember the Tacoma Narrows bridge - it had a factor of safety of more than four times max expected load - but twisted in light winds and fell into the water under no load at all. But up until the day it failed - the highway department could claim it was safe and well designed.

It failed because it was subjected to different stresses from different causes way beyond any expected by its designers, in twisting weird ways its designers did not expect, even though it was under less simple deadweight loads than anybody planned for.

But it failed nonetheless. All I need to do to "predict" failure is recall my 10 years experience in computer SYSTEMS-Level testing and services and systems (such as power plant construction, fueling, and operation.) It is extremely difficult to get a program to work under perfect conditions - and by review alone of QA papers on software testing, I expect 7-15% of the remediated programs to fail in NEW ways because of the revised coding entered - and can expect 20-45% of any unremediated program in an unremediated system to fail do to unknown miscellaneous causes.

It isn't the single failure of a single chip in a single power plant that concerns me - that is your experience, and I congratulate you on doing even that much. I'm looking at 4000 power plants - each of which just to be remediated needs more time than we have available - if 40 are in systems testing that implies that only 1% are tested. And I have my doubts about the adequacy of even that level of testing - again, because I have spent 10 years debugging revised software when it "previously worked fine" but failed after changes.

Life isn't simple. It isn't one system. One chip.

You mentioned "the survivability" of one cockroach elsewhere - good example - now, trying to remove enough bugs from the world's software to maintain today's systems is like trying to kill all the cockroaches in your house. And being flogged for each found.

The ones you miss may not kill you, but they will be painful.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 01, 1999.

Thanks Norm - I needed that (the music I mean.) It's really depressing out here this last week, I agree with Greybear - Y2K sort of pales at the moment facing possible WWIII.

-- Sylvia (home@wood.com), April 01, 1999.