The Arizona Republic recently put out a 7 page section of their paper devoted only to Y2K. All the stories were positive about the effects of Y2K. One story in particular was about the Salt River Project near St. John's, Arizona. In it they state: "On Jan. 8 SRP cranked forward the clocks at the 700 megawatt coal-fired power plant to test its Y2K readiness.... It continued to generate electricity flawlessly without any glitches.... SRP also tested the system for its functionality during leap year.... Again the system functioned perfectly as it did when 16 subsequent potential problem dates were tested.... The plant operated flawlessly when the clocks were moved back to the present."

Now this isn't the first such story I've heard about no problems with testing. Another thing I'm hearing is that the embedded chips are turning out to be a non-problem. Can someone with expertise on this shed some light? I realize that there is a lot of disinformation being pushed down the public's throats. If this is more disinformation, they have taken it to new heights. Confused in San Diego.

-- Anonymous, February 09, 1999

Answers

Confused,

You mention 2 facts of interest. 1. SRP unit rolled over with not glitches. 2. You have heard this type of success story before.

Read back posts here. I mentioned that in addition to a Canadian unit that I BELIEVED that 2 other units were rolled over in the US and at least one was still operating that way. I cannot and will not disclose names (same policy as Rick Cowles). Nobody seemed to care or believe or take notice.

In your own post, you note the good new, mention others like it, and then go on question the honesty and integrity of the utility that disclosed the results. This is the problem in my mind. What incentive do I (or anyone else for that matter) have for spending my free time here relating results that are either blown off as fabrications or questioned for technical accuracy/effectiveness?

By the way, I have also posted here that my opinion is that observed test results show no embedded problems that would prevent a device from being Y2k READY. I have no problems with others considering my personal opinion as disinformation, but I certainly wouldn't blame anyone for seeing disclosure as an exercise in futility.

-- Anonymous, February 09, 1999

CL, there's no need for the self-righteous indignation. It appears that Confused was merely asking for corroborating information about SRP's successes, or information to dispute them. And one is certainly justified in being confused. On the one hand, credible sources predict at least "localized" outages due to embedded systems, or worse; on the other hand, the NERC report and the SRP article state that to date, embedded systems have not been found to be serious threat to power generation.

One caveat: a power "unit", as I understand it, is one generator, not to be confused with a complete power generation plant.

Another caveat: power must also be transmitted and distributed, fuel and communications must be availble, etc.

And why not disclose company names? Isn't this information public? When I hear *anyone* say, "I believe two other units...", or "a Canadian unit...", I put the statements in the category of, "Once upon a time..."-- until verified. We need information which is more specific, not more vague.

-- Anonymous, February 09, 1999

CL, there's no need for the self-righteous indignation. It appears that Confused was merely asking for corroborating information about SRP's successes, or information to dispute them.

* Think about what you are saying. How in the world would ANYONE beyond SRP have the ability to corroborate the test results of their integrated unit testing? If you are waiting to corroborate from an independent source, dont hold your breath  its impossible. The underlying problem is an unwillingness of many here to accept quantified test results - with a corresponding eagerness/willingness to accept skepticism based on conjecture.*

And one is certainly justified in being confused. On the one hand, credible sources predict at least "localized" outages due to embedded systems, or worse; on the other hand, the NERC report and the SRP article state that to date, embedded systems have not been found to be serious threat to power generation.

* Please place the same requirements on the skeptics that you do on the utilities. NERC has provided raw data to support their conclusions. The specifics of the detailed test procedures used may legitimately be questioned in regards the NERC report, but the raw data is provided (I have stated one example of an embedded device where test details are available from the vendor on the web). Who did the testing on utility embedded devices that yielded mission critical failures? Specifically, which embedded devices failed? (I would appreciate this info so that I can remediate the problems). Where is their raw data? If there is none, then the prediction of scattered outages is unfounded speculation and should carry the same weight as Punxy Phils prognostications.*

One caveat: a power "unit", as I understand it, is one generator, not to be confused with a complete power generation plant.

* Your observation on a unit is correct. Units are usually tested when they are not on-line (proper prudence). It may be extremely difficult to get more than one unit off-line at a time and not impact production requirements. Units are started, stopped and regulated discreetly, so I really do not think your point is of any consequence at any rate.*

Another caveat: power must also be transmitted and distributed, fuel and communications must be availble, etc.

* Yes, and fuel and communications contingencies are developed and are being tested. T&D aspect was addressed above. Show ME the DATA. It is rare to see instances where predictions of outages (of varying scale) are subjected scrutiny and questioned for their accuracy. This is intellectually dishonest. The same test should apply to skeptics of the utilities and NERC. At least the SRP conducted quantifiable tests. Where are the test results showing T&D embedded device failures. I challenge ALL to give me one example! *

And why not disclose company names? Isn't this information public? When I hear *anyone* say, "I believe two other units...", or "a Canadian unit...", I put the statements in the category of, "Once upon a time..."-- until verified. We need information which is more specific, not more vague.

* Rick Cowles has adequately address why HE cannot divulge company names. Search this site and you will find his post. Same applies for me. My opinions are PERSONAL and cannot and do not reflect the opinions or positions of any company, government agency real or imagined. Certain forums for info sharing are only effective if proprietary information is guaranteed protection ( utilities gather and share info from competing vendors  ethics require that the valuable info being shared does not create a competitive disadvantage to the vendors willing to share).

-- Obed Salazar (r26826@email.sps.mot.com), February 09, 1999.

-- Anonymous, February 09, 1999

CL: Were these isolated occurrences?:

Real Life Y2K in the Industry

What are we to make of this evidence? Does this not expose significant risk?

-- Anonymous, February 09, 1999

TVA: 8 Units http://www.tva.gov/y2k/progress.htm

Avista: 10 Units http://www.wwpco.com/y2k/y2k_qna.asp

Southern Co.: 12 Units (8 more by May 1) ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/southern co presentation.ppt (note: PowerPoint Presentation)

Ontario Hydro: 4 Units ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/ontario.pdf (note: Adobe Acrobat File)

TransAlta: 12 Units ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/transalta.pdf (note: Adobe Acrobat File)

-- Anonymous, February 09, 1999

Did these 46 units remediated, or did they never have a problem with Y2K in the first place... What percentage of the total number of units does this represent? Will this number of units be able to sustain the grid?

For some reason the orginal link didn't take. Here is a link to the euy2k site that had some real life examples of failure that I have been attempting to prove or disprove as a reason for continued risk: http://www.euy2k.com/reallife.htm

-- Anonymous, February 09, 1999

Troy,

Thanks! I enjoyed the page here describing failures. This is the kind of response I am eager to see. Good job.

Now to debate the issues there. Look close at the examples.

HECO article was based on a speculation of a "possibility of a problem" and a programmers guess of "probable" problems. Notice that they never tested the old EMS applications and so there is no confirmation that a problem actually existed. (I don't blame them, you don't test a legacy system that you plan to replace). The HECO article did not explicitly state any testing on the new system (it MIGHT be implied that the system was OK). They did mention the OS was compliant, but I read that to mean that the OS was also OK on the old system.

The Western Power article is pure, unadulterated speculation. As a matter of fact, it is more of a dissertation on consequences than it is a factual description of measured problems. We all agree on the consequences, I just see no hard data to support an assumption that the problem will manifest.

The ITRON article is about a meter that will not disrupt power even if it fails catastrophically.

The British article is the only one of the three that has any meat to it. I hope they are working hard to remediate the problem. That is why I am testing. I just have not encountered anything of this serious nature.

Thanks for the reasoned reply. Do you disagree with these comments?

-- Anonymous, February 09, 1999

I agree with CL: the same standards of evidence should be applied to optimistic thinking as to pessimistic thinking. Personally, I prefer to err on the side of cautiousness. My photo-Voltaic system will eventually pay for itself, regardless. Especially if electric prices rise.

As to the challenge to find evidence of the impact that non-compliant embedded systems can have to electric service, I found the following sources:

http://www.zdnet.com/zdy2k/1998/11/5239.html#table3 Table 3 shows that the Seabrook Nuclear Power Plant found 3 defects with a Safety Implication, 13 cases with a Plant Trip implication, and 5 cases which would cause Generation Reduction.

The NERC report, on page 21, states, "Testing of non-nuclear generators continues to indicate a minimal number of failures that might cause an unremediated unit to trip". Then, "There have been a few instances during Y2K testing in which it is thought that a lockup of an unremediated DCS system might have caused the unit to trip".

These two statements, and the Seabrook report above, seem to be contradicted by the NERC report's statement on page ii, "the types of impacts found thus far...do not appear to affect the ability to keep generators and power delivery facilities in service and electricity supplied to customers".

I know that CL is asking for instances in transmission & distribution; I don't know whether some of the cases above are in T&D, or if they are all in the generation process.

-- Anonymous, February 09, 1999

Confused in San Diego, there is a discussion outline on generation-testing-strategies for a recent electric industry conference at:

ftp://www.nerc.com/pub/sys/all_updl/docs/y2k/generation-testing-strategies.pdf

(You need an Adobe Acrobat Reader to access this.)

Here are some sections of that outline:

"A number of organizations have reported successfully performing integrated rollovers of on-line power plants.

1. What have we learned from these rollovers? * Able to leave in forward time and have not had any problems. * Rolled forward remediated units * Vendor support may be an issue (due to legal concerns) * Provides public relations advantage * Provides as much of an integrated test as possible *Allows us to mitigate risk *After a while it may not be a test but a risk management decision *Sustained testing provides additional assurance *The risk may now be reduced because more companies have done this test. 2. Does the existance of embedded chips within a power plant limit the ability to complete a full rollover of all systems? Is our confidence in the results diminished by the inability to rollover some devices? What can be done to mitigate this concern?

*This seems to be the fear * Our confidence is diminished for the ones that you can't get to *Other methods of assurance mey be taken such as reviewing the code (ladder logic in PLCs) *The concern can be mitigated with contingency plans

5. Given the experience to date with integrated rollovers of on-line power plants, and weighing the complexity, costs, and resks associated with such tests, what guidelines are appropriate for the industry with respect to continuing further testing of this type? * It is difficult to identify any additional cost after the remediation * Some of the systems are so integrated with work management and other applications that it is difficult to roll-forward and leave there. A bridge might be built to help with this issue. *There are two different but related issues -- perform rollover testing and leave them in a rolled forward state * The major costs are in the actual remediation of the units, minor costs to roll forward *Guidelines: *Remediate first *test units that provide a representative sample, depending on the size of the company and political invironment"

This outline seems to indicate the the electric industry is asking themselves some of the same questions which are being asked on this forum, and that there is not confidence that embedded systems can be part of a rollover.

-- Anonymous, February 09, 1999

Obed,

EXCELLENT POST!!! Yes, I am involved in T&D with some pre-Y2K experience in power stations (Nuke & F&H). You are correct, the NERC report does say some failures would trip units if not remediated. These are few in number, and hopefully my comrades in Gen. are addressing them. An excellent example would be the British example listed on the Real Life Failures page of this site. T&D failures are rare (almost to non-existant).

The Seabrook Report is an excellent reference. You and Jon both mentioned it. It is hard for me to decipher with a utility background and Y2K testing time. I have a new appreciation for the challanges you have been facing in deciphering this. Bottom line - an analysis of the Seabrook report deserves its own thread. Hope Rick or some of you can take a whack at it, I won't have time until next weekend.

My initial reaction to Seabrook is that the Tables that seem to show failures do not imply that any testing has taken place OR represent any test results. I believe that the only conclusion that can be drawn from Tables 3 & 4 is that they could not get confirmation from the vendor that the systems were compliant. I am fairly certain that the last column (showing consequence of failure) will not be the automatic result if the item is not fixed. ILLUSTRATION: A DFR was listed with a strategy of FIX Y2k testing required (or something like that). The last column stated REDUCED GENERATION. A digital fault recorder has no outputs that would initiate reduced unit output, and provides no indication to a plan operator that would cause him to manually reduce output. At worst, IF a unit trip were to occur, it MIGHT take longer to diagnose the problem and bring the unit back.

At any rate, the Seabrook report deserves a more thorough treatment. For now, please do not assume that there were problems found during testing - I really don't believe that is a correct interpretation of the report.

Thanks for the positive dialogue and excellent post.

-- Anonymous, February 09, 1999

cl: Are you suggesting that this report was based on a query of manufacturers and not on actual testing?

-- Anonymous, February 10, 1999

Bonnie: I know that EMS is Energy Management System. What is DCS?

-- Anonymous, February 10, 1999

Troy, this definition is compliments of Jim Lynes:

The system at the generation station is typically called a Distributed Control System(DCS). As the name implies the intelligence is usually distributed across the plant and tied together with a fiber LAN. DCS systems perform closed loop control of fast processes. Sensors are scanned from .01 to 1 second cycles. The operator responds to alarms and may adjust set points. He usually doesn't control the process directly.

My opinion is there is more risks with the DCS systems than with the SCADA systems. I manage one of each type of system and I will try to answer any followup questions that you might have.

Jim

-- Jim Lynes (james_lynes@corp.disney.com), December 03, 1998.

-- Anonymous, February 10, 1999