Which is it?

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

For the sake of discussion, let's consider it a given that what CL and FactFinder report is completely true. That there are no embedded systems with Y2K problems which will affect the generation, transmission, or distribution of electric power. It's been postulated that because of this, preparing for power outages is unnecessary and those who are promoting preparation for power outages are ill-informed.

Yet we also have had Carlos asking about workshops or seminars on Y2K, FactFinder states some kind of concern about the late start the industry got, the quality of work being done, and the impact of Y2K bugs in regulatory software and equipment, Menno writes about proper testing protocols, CL said he (she?) continues to work hard on the Y2K project, NERC has asked some utilities to "accelerate" the work on their projects, and has scheduled nationwide contingency planning drills.

Why? If no one has found any problems which will interfere with getting electricity to my house, and therefore I don't need to prepare for any power outages, then why are Year 2000 projects in the industry still ongoing? The money being spent from this point on (and all spent prior to this) will most certainly be passed along to customers eventually. Is continuing Y2K projects and worrying about deadlines just a ruse to raise our utility bills? And our tax dollars certainly shoudn't be wasted on national drills, or NERC reporting efforts, if there is no danger of outages.

Either there are Y2K problems which *must* be professionally addressed by a deadline, and which *can* impact the electric industry's ability to provide me with reliable power (regardless of the embedded systems non-problem) or there is a hoax being perpetrated on the American public far greater than anything a "doomster" could think of.

Which is it?

-- Anonymous, February 24, 1999

Answers

No matter who is right; HOW can we possibly believe that all the appropriate work will be acomplished in time. For example: when someone like myself (who lives in New Jersey) reads that both Conectiv and PSE&G, are presently only 10-15% into repairs... why should we trust anyone?

New Jersey is the most densely populated state, and these two carriers -that cover most of it- are so far behind in repairs??

-- Anonymous, February 24, 1999


Bonnie, that's what I had asked in an earlier post. Why all the work,worry and money spent if CL and Factfinder are correct? Assuming they are right, why not stop all operations right now, say oops.... just kidding...guess we were wrong about the embedded systems and everything else and go back to normal routines. Why have any more testing? Are the electric companies doing all this work because they need a little excitement or are there really problems? I'm not discounting what CL and Factfinder have said. I have no reason not to believe them. So, if they are right, then stop the testing...it's a waste of time.

-- Anonymous, February 24, 1999

Thanks for this post. This is what I was trying to ask under my "doom and gloom" question, but you put it much more specific. I would like to have this question answered if possible.

Thanks again.

-- Anonymous, February 24, 1999


E.Z., as B.Clark has also noted, if there are no Y2K problems which will affect the reliability of electric power, then it doesn't matter how far utilities are in their progress. They can stop their projects now and the power will still flow in 2000. That's the point to my question, "Which is it?" Embedded systems aside, either there is still the potential for power disruption or we don't have to worry about project deadlines anymore. If we do still have to worry about utilities finishing work in time, then I completely agree with you that there is plenty of concern.

It's just time we cut to the chase. There are either problems outside of the embedded systems issue, or there aren't. It can't be both ways. If there are other potential problems, then preparation is still wise and prudent. If there aren't, then the Y2K project time and expense is not now warranted and consumers shouldn't have to foot any more project "repair" bills.

-- Anonymous, February 24, 1999


Bonnie, you know as well as I do, that Anyone who doesn't prepare is a fool; when one weighs all the other potentially *emormous* problems... Like no foreign oil, etc.

Even if the electrical grid manages to stay -relatively- intact...(which appears to be an *unanswerable* issue, considering all of the complex interdependancies)...there are so many other things that are going to fail miserably...

Not to mention the Possibility of chemical explosions, fire storms, industrial emergencies, and countless 'smaller' situations. What's that old Arabic saying? "Trust in God, but tie your camel"?

-- Anonymous, February 24, 1999



Travel back with me, a few months...

CL, FF, Troy, Menno, Carlos; you might find this particular post instructive in terms of what regular folks are struggling with.

Given the nature of this current thread, everyone might want take a moment to reflect on this post from November (and the followups):

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000G Bh

-- Anonymous, February 24, 1999


Here's my 2 cents worth. I am up to my neck in Y2k at a fairly substantial utility in this region. Monday was a meeting of my company's management about Y2k contingency planning, yesterday was a conference call with DOE's Y2k Program Manager discussing the April Drill, today a call with regional communications people to contingency plan, tomorrow I'll be chairing a regional utility Y2k Steering group, Friday I'm meeting with my Drill team. Every week is the same tempo. I only started my job in early November. Little if anything was really going on before that. The utilities were slow to get out of the denial stage. That appears to be a solid fact. NERC and people like you have changed that. That is good in my opinion.

Another fact is that few problems have been found since we woke up. But that doesn't mean that we stop looking. Problems have been found in thermal powerplants that need to be fixed. Have all been fixed? Probably not, but in this region it is my mission to get them done, get the plants tested and rolled over. Other problems have been found (phone PBXs, control center security systems and HVAC controls) and they are being fixed. I am pushing for more agressive on-line testing of all sorts of systems. I do not take Y2k lightly. Should I stop and quit spending money and time on Y2k? I am not yet satisfied personally. I like having power as much as anyone and don't know the answer to the big question anymore than you do.

-- Anonymous, February 24, 1999


The answer was, is and will continue to be this: many embedded systems have real Y2K problems; most power plants use embedded systems; some power plants are guaranteed to have problems; some power plants will have severe problems.

Just the facts: starting around 1996 there have been roughly 4 billion microcontrollers (MCUs) shipped _each_ year. About 1 billion 4-bit MCUs, 2 billion 8-bit MCUs, 800 million 16-bit MCUs, and 200 million 32-bit and greater MCUs (if you stick a 300 Mhz Pentium single-board computer running Y2K bug-ridden Windows 95/98/NT into a telephone switch it's still an embedded system, albeit a pretty ugly one).

Most MCUs that are susceptible to the Y2K bug will be at least of the 16-bit variety. Well that gets us down to only 1 billion. Phew! Now let's suppose that only 1% of the embedded system designs had real Y2K problems. That's 10 million embedded systems shipping each year with Y2K problems ready to trigger.

Now since most everybody didn't really get clued into the embedded systems problem until the middle of 1997, and the ones who cared didn't get their products fixed until the end of 1998, I'd say another 10 million bug ridden systems got shipped in 1998 too. I also do not hesitate to add that many companies still haven't fixed there embedded systems designs -- they're still in the "Do we _really_ have a problem with this Y2K thing?" stage -- so we can expect that Y2K bug ridden products will be shipping well into 2000.

Don't let anyone try and tell you there is no such thing as an embedded chip with a Y2K problem. Just say to them, "Dear Madam/Sir, with all due respect you don't know what you are talking about, I encourage you to visit this web page: www.mot-sps.com/y2k/rtcalert.html, read it carefully and then we can converse further." (and say it with a smile)

My favorite line from that particular page: "The following products are "NOT Year 2000 ready and a fix is NOT planned. A resolution (replacement, upgrade) will NOT be provided" per our Year 2000 Status Definitions."

There are seventy (70) distinct chips listed on that page. I encourage you to ask Motorola how many millions of products use those chips.

Keep in mind that Motorola isn't the only company that makes such devices.

If you doubt my numbers for embedded microcontrollers/microprocessors I leave you with this one reference for one chip family from one manufacturer: http://www.microchip.com/10/Edit/pRelease/PR72/index.htm

Best regards, --AJ

-- Anonymous, February 24, 1999


Bonnie, Rick, et al

Yes I understand your frustration. If I understand the questions they are:

"Why can't you give me 100% assurance that nothing will happen on Y2k?" "If you are so confident why bother testing any further at all?" "Given this confusion, how should I prepart?"

Questions 1&2 are related. If anyone has taken a debating class, you will no that it is impossible to prove a "negative". This is the crux of why I issue no assurances, just truthful observations I have made in the course of thorough testing. If I am a pilot and perform a thorough pre-fligt check, can I give 100% assurance that we will take off safely and "arrive alive"? I can not intellectually rule out the infinite number of possible exceptions. What I can and will do is tell you that my tests have uncovered little at all, and nothing of consequence.

As for question #2, engineers in general, and utility engineers in particular are conservative. They make Dilbert look like Dennis Rodman. If I finish my critical testing ahead of time (I will), I will go on to test non-critical stuff. If I finish non-critical stuff (I might) and the contingency planning and practice drills (I must), THEN I will go back to the critical stuff and test again in a paranoid frenzy attempting more and more absurd hypotheticals until the rollover comes (in attempte to systematically identify and test the infinite), or my pointy haired bosses put me back on the duties of my previous (and more fulfilling) life.

Utilities got a slow start. When I assumed this role, my predecessors primary accomplishment was attending meetings. I am now nearing completion of my embedded systems testing. I see my peers in the nuke and F&H stations doing the same. They have more money than I do, have replaced major systems. I suspect they may even have used the hype of Y2K to get "nice to have" across the board upgrades to their DCS system at each and every plant. I don't know for sure, but I sure can't rule it out. We will be ready unless my peers have been totally incompetant. Based on our excellent nuclear performance, very good F&H performance, and a very reliable T&D system (blush, blush) I don't forsee any problems with my major utility.

Are some utilities behind me? No doubt. Is it a problem? If they are behind testing the devices I have tested - no. If they are behind on systems with "home-grown" software like DCS and SCADA master (items which open the possiblity for a programmer to build his own Y2K bug), then maybe some problems. My attendance at EPRI nuke and F&H session lead me to believe the major utils are on the ball.

Bonnie, you ask a question I have wanted to pose myself. Are you bringing up Y2K costs to ratepayers and taxpayers in order to deduce there MUST be a problem? OR is the growing sense of calm and reason that points to a lack of problems calling you to truly question who will pay? Think back to last Oct/Nov. Read the posts here calling for government takeover of the industry - calls for a purely socialistic energy sector. They all question why why why aren't spending more, testing everything - my o my what is this type testing??? When competition comes and the utilities that went overboard and did high sampling rate tests go to pass the costs on, will you still applaud them and stay loyal - or high tail it to those who elected type testing (we will welcome your business) and have lower rates???

That is one of my peeves about this whole situation. Accountability. If, after a sound analysis of the credibiltiy and weight of argument brough by FactFinder, Menlo, Art, - you decide to heed instead the prophets of gloom, what is your cost? A sheepish attempt to return an unused generator and a month or so of pancakes and canned beans 3 meals a day? You can shrug it off with "better safe than sorry" and hope the neighbors don't ask about the drums of water in the back yard.

If I'm wrong, I lose my job and in about 6 mos, my house, car and yacht (yea right - on a utility salary). That is the other reason I will continue to test, my life depends on me being right. If I'm wrong, my lights will be out like yours, but I won't have a job. And by the way, I would venture to say that just attempting to provide some light in this forum could have the same result - we all risk much just being here. But then again, every single day engineers and technicians are out there doing maintenance testing and op checking that counts on their competence to keep from putting lights out. You never know or see it, unless they make a mistake. Maybe after this is all over, Rick could come here and explain the dangers of testing breaker failure and transfer-trip schemes. No, you'd never sleep.

Well, whining isn't very dignified. I didn't proof read this so it probably rambles and serves only to use up bandwith and harddrive space while effective avoid addressing any real questions. Sorry. How many bestselling books are written by engineers? Dilbert doesn't count.

Domimus Vobiscum

-- Anonymous, February 24, 1999


CL, I personally think utilities should be making their Y2K projects top priority, spend all the money needed, and if my bill has to go up later, fine. One of the points you and FactFinder have been making on this board in the last weeks is that you consider preparing for power outages unnecessary, and your opinion of those who are preparing has been, shall we say..rather disdainful? Your findings of no embedded systems problems have been used to support an overall no problem scenario.

The question I posted was an attempt to delineate the concept that there are other Year 2000 problems in the industry, besides embedded systems, which might disrupt power delivery if not discovered and properly fixed in time. I submit this is the case, but I was waiting for you to either admit it is, or if you really don't think there are other problems, then to tell me why you're not suggesting to your managers that the projects cease now and save everybody the time and trouble. You may have done it in a roundabout way, but you did admit to "maybe some problems" in certain cases.

Bottom line here. If there *are* other potential problems besides the embedded systems you talk about, then you're not only betting that you're right in your testing, but that everybody in every area of the industry will also be right in their findings for every area and everyone will also be competent and on time in their repairs. I'm not willing to make such a huge bet. I believe it's both logical and wise to be prepared for disruptions, and that choosing such a path is in no way ridiculous or imprudent -- even for those who work at utilities. I think Chuck was very honest when he said he didn't know the answer to the big question either. Many of us here have indicated willingness to accept your findings of no embedded systems problems in your utility. We're happy about that, we're thrilled there are conscientious individuals working on the problem, but we're also trying to get you to see why preparation in the face of uncertainty in many other areas is not stupid. I don't know how I can make the situation any clearer. Since both NERC and the U.S. Senate have admitted to the possibility of local outages, and we don't know where those outages will be, then can't you give us some credit and agree that being prepared is not a ridiculous decision?

-- Anonymous, February 24, 1999



Bonnie,

When there are y2k-projects the chance there will be an local outage could be a little bit higher or lower than normal. It depends on the y2k-readiness and the contingency plans. When you do nothing the chance is much higher. So take a look at the y2k-projects in your neighbourhood and decide than to make preparations or not.

-- Anonymous, February 25, 1999


For the sake of this discussion, here is a radical thought. Let's say "All utilitiy companies in the nation achive 100% compliance in time", they are still toast. It is a recognized fact that any one working on a Y2k project tends to look at the the problem as a whole based on his experience with his own project. If some folks in this discussion are working on a project that is moving ahead, thats wonderfull. However, this does not appear to be the case nationwide. Back to my premise: "All utilities magically become 100% compliant overnight". It is obvious whatever type of generation plant is being considered, (coal, fuel oil, etc.) in order to run the generators (with the possible exception of Hydro) fuel is required. What is the state of the fuel processing and distribution? I think it is safe to say that assuming at some point in time the nuclear plants go offline as a precautionary measure that this will leave a tremendous burden on the plants using fuel oil and coal to operate their generators. We know that the ports and large tankers have already reported as an industry that "they will not achieve compliance, and have ceased to try. Cheveron has reported the same. Koskinen himself suggested in a report recently that the United States could lose up to 70% of it's oil refineries to embedded systems. Burlington Northern railroad carries 80% of the coal that is transported around this country and they did not start on their substantial Y2k project until mid 97, a little to late in my book. All of the railroads switches across the country are automatic, manual switches are gone. It is obvious that even if fuel continues to move at all (which may not even be the case) a tremendous slow down is unavoidable. Some of the largest utility companies use a massive amount of fuel in a normal day. A severe shortage would cause a significant decrease in generating capacity. Reserve fuel is not enough to rely on as the normal company does not have more than about 3 weeks worth. Fuel will be tough to get! Without fuel, even a compliant company cannot generate anything!

Ok, now let's look at the truth. The truth is all of the utility companies in this country are not even close to 100% compliant. Instead of a few missing the mark and not making it, it will be a very few that DO make it. Of course some will, but most will not. Y2k non-complince will shut many down. Fuel shortages will shut down the rest, or at least severly hamper generation capability.

-- Anonymous, February 25, 1999


Bonnie,

You are to be commended on several fronts. One is for your objectivity and willingness to point out that there is no rational motive for NERC or Utility folks to be deceptive (or others to assume as much), but that lawyers and public relations/media types naturally speak in "company" terms.

The other is you tireless efforts in researching this. Clinton ought to be glad you weren't on the Kenn Starr's staff - he'd be toast by now.

I must agree that recent statements by NERC and the Senate could cause a reasonble person to conclude that preparations should be made. These statements are presented as "we can't rule out..." or "we predict no greater than...". As stated, they can be (and are) logically interpreted to mean that something (usually stated as "temporary, local or regional outages") WILL happen. So in light of these statements, I cannot characterize you or your preparations as stupid or ridiculous.

I personally question whether these statements reflect the proper response to a measured engineering problem, or the regrettable influence of litigation fearing lawyers and media/PR specialists. You see, I view the lawyers as being orders of magnitude more reactionary than those of you preparing for calamity. THEY are the ones keeping utilities and vendors from freely sharing info. with each other and the public. This contributes to your frustration and fear, inhibits cooperation, and leverages the lawyers as the only true winners in this.

I cannot say that there are not problems at smaller privately owned generation plants. What I see at EPRI leads me to believe that the big utilities will all have their plants on-line and makin money. If there are problems at plants, will they shut the plant down, or just reduce environmental montioring? Don't know. Can the system survive without the co-gens? Ours can. We've never counted on them to meet base load. Conceeding the possibility of uncorrected problems does not necessarily equate to customer outages. If your area is not supplied by generation from a major utility with an active Y2K program and EPRI participation, then maybe you should prepare in the event that weather precludes making up the difference with power import from purchases. (Yes, I believe the T&D system will be intact for such transactions).

My concern is that the bias here that lends credibility to doomsayers and requires critical analysis of NERC and Utility "good news", in combination with NERC/Senate predictions could lead some to make preparations that are not proportionate to the threat. People of modest means overspending from their needs rather than their abundances. I would like to see those who predict localized, temp outages give the details of what devices will trigger these outages.

Bottom line here. If there *are* other potential problems besides the embedded systems you talk about, then you're not only betting that you're right in your testing, but that everybody in every area of the industry will also be right in their findings for every area and everyone will also be competent and on time in their repairs. I'm not willing to make such a huge bet. I believe it's both logical and wise to be prepared for disruptions, and that choosing such a path is in no way ridiculous or imprudent -- even for those who work at utilities. I think Chuck was very honest when he said he didn't know the answer to the big question either. Many of us here have indicated willingness to accept your findings of no embedded systems problems in your utility. We're happy about that, we're thrilled there are conscientious individuals working on the problem, but we're also trying to get you to see why preparation in the face of uncertainty in many other areas is not stupid. I don't know how I can make the situation any clearer. Since both NERC and the U.S. Senate have admitted to the possibility of local outages, and we don't know where those outages will be, then can't you give us some credit and agree that being prepared is not a ridiculous decision?

-- Anonymous, February 25, 1999


What we need the utilities to say that their plants, distributions and 5000+ suppliers are all compliant and they guarentee that there will be be no outages or few outages or whatever.

My company, Alternate Energy, can make that assurance about the products we sell. I know the power and heat will stay on at my house for more than 3 years. We have had a combined solar/wind/ microcogeneration plant since 1991.

I am optimistic about the future after y2k. Advances in solar panels, fuel cells, microturbines, etc. will mean more and more people can be affordably independent.

-- Anonymous, February 25, 1999


Cl, Sorry about this being so long.... I don't particularly trust people who don't give their names like everyone else on this Board. You say that you cannot because it might jeopardize yourself but you are saying everything is fine,peachy-keen, how can that jeopardize you? I'm sorry I just don't understand that reasoning. You are like all the other people who talk about this situation from a defendants point of view. You state everything is fine...TRUST ME!!! I know, I work here... That does not give me a warm fuzzy, sorry...

To your message.. Your comments are in quotations...

You state... "Utilities got a slow start." Why.. I understand that this issue has been known in the federal government for over ten years.. why did evryone finally "get it" in the past two or three years? That is negligence. Should we trust our lives on any industry who is so ignorant or negligent as to not test ten years ago by rolling the clocks forward on a power plant that has been taken off line for maintenance??

You state... "When I assumed this role, my predecessors primary accomplishment was attending meetings." Should we believe that every other person in the industry EXCEPT that person you replaced was as conscientious as you are?? I feel that I am a very competent person, but there is no way that I would trust even 10% of my fellow employees to do a specific job as well as I could. And yet you expect us to trust that there are thousands of people who have enough integrity, honesty, and skills to do a Y2K remediation and testing job as well as you. There are a bunch of highly educated persons who read this group, those I trust, but there are a lot of people out there that I do not trust.

You state... "We will be ready unless my peers have been totally incompetant." And yet you just stated that you replaced a "Peer" that was totally incompetent.

You state... "I don't forsee any problems with my major utility." That's great what one do you work for so that I can move to that area? BUT, what about the company that that other person just moved to... Are you so confident about that company?

You state... " Are some utilities behind me? No doubt." That's comforting to know, especially if someone I know and love is undergoing emergency surgery when the power goes off in that "local area". Or the life saving medicine has just spoiled because the refrigeration has been off for five days.

You state... "Is it a problem? If they are behind testing the devices I have tested - no. If they are behind on systems with "home-grown" software like DCS and SCADA master (items which open the possiblity for a programmer to build his own Y2K bug), then maybe some problems." Another nice to know unless I live in that "local Area". I live in the North East... what happens if we have another Ice Storm around the turn of the century. They fix the lines and think everything is OK, but there is no power because we are in one of those "local Areas".

You state... "Yes I understand your frustration." I'm sorry but... NO YOU DON'T. You at least have access to information about the status of the industry, you (probably live around your utility) and "know" your power will be on, since you are making sure of it, You have the position to be able to look at your coworkers and be able to trust that they are conscientious people, with integrity, who know what they are doing and will try to do their best. And no I am not saying that all electrical industry people are bad empolyees but I am saying that there is that 10%. The people in the industry are probably working their butts off, doing the best job that they can, and we all owe a HUGE debt of gratitude to them, BUT I do know what management is like and how they solely look at the bottom line. So please you don't understand our frustration. I think the government should freeze all liabilities in order to get the information out, but heh, who am I.

You state... "I cannot say that there are not problems at smaller privately owned generation plants." How many privately owned plants are there thatn large ones like the one you work at. What about all of those people..;. Should they prepare???

You state... "My concern is that the bias here that lends credibility to doomsayers and requires critical analysis of NERC and Utility "good news", in combination with NERC/Senate predictions could lead some to make preparations that are not proportionate to the threat. People of modest means overspending from their needs rather than their abundances." If the companies did what they should have done the fixing and testing would have been done five years ago. But since they were negligent, the population now has to pay for it. You say that preparations are not proportionate to the threat... and yet you state you don't know whether the small companies will make it. What if a small company goes belly up and can't produce.. what happens to that community?? What happens if the area goes through another ice storm and is out for weeks, No threat huh?? What happens to one of the local hospitals or nursing homes when their "local area" experiences an outage??

You state... "I would like to see those who predict localized, temp outages give the details of what devices will trigger these outages." See below paragraph about transportation.... or do you believe other parts of the world will experience no problems too???

If you will give me a list of applications you have tested, the independant company who verified your results, the methodologies you used, the list of suppliers you have, a copy of their independently verified upgrades, your status as to their implementation, your suppliers suppliers, your list of backup suppliers if those go bankrupt or can't produce, the county you have purchased for your stock piles of fuel, gasoline, and deisel for your generating plants, your black start capability, your contingency plan to get spare parts and fuel to your site when and if the trains can't move(slow movement due to loss of computers tracking and switching capability), trucks don't move (because the middle east can't produce, refine, or send petroleum products here and we can't receive or distibute them), planes don't fly(because no insurance company will insure them, the ATC system has to go back to paper rather than RADAR, navigation /aids instruments don't work due to a "local area" outage), the telecommunications you use to monitor and regulate energy passing through the lines get a busy signal, and Banks can't pay for any of it.

Please state your area of expertise that you feel you are qualified to determine whether these areas will function properly...

One final thing... Exactly how many "Local area" outages, or those "small generation plants" does it take to bring down an electrical region?? I could go on but I think you get the idea..

CL, We are not doubting your or your co-workers expertise or motivation. We ARE doubting whether there is enough time to fix everything that needs to be fixed in order for our society to function as normal. We have been lied to too many times to believe what anybody says that uses PR flacks or lawyers to say it...

I don't believe the FAA when in November they were 99% compliant and now they say they are 90%, that means they have no clue. I don't believe the DOD when they were caught lying by their own Internal investigators, I don't believe the federal government when all they do is REDUCE the number of "mission Critical" applications in order to make their numbers look better. I do believe GM when the new boss said he tried to roll forward a plant and it shutdown. And I certainly do believe that there will be NON-MISSION critical applications that will be found to be Mission critical. And I believe that those same mission critical systems that haven't even been looked at yet will have enough impact to bankrupt many many companies, possibly yours and mine. Sorry about babbling.



-- Anonymous, February 25, 1999



Sorry that last sentence should read"..And I believe that those same NON mission critical systems that haven't even been looked at yet will have enough impact to bankrupt many many companies, possibly yours and mine. Sorry about babbling.. Again sorry about babbling..

-- Anonymous, February 25, 1999

CL, thank you for trying to see things from the perspective of a non-industry individual. I'm also trying hard to incorporate your perspective into the body of knowledge I've accumulated. It may be very difficult for any of us to stand in another person's shoes and attempt to see through their eyes, but I think it's always worth the effort. You're a gutsy person who has endured in presenting your views, and that's admirable.

For the sake of injecting a little humor here, there *is* something almost all of us have experienced in the Y2K arena and can easily identify with each other about. That's being "put upon" by those with opposing views! Some of us have even been accosted by both sides at the same time, on the same issue. _laughing_ We're all just doing the best we can to figure things out and it AIN'T EASY! Not to mention the day I discover two people who agree on everything, I'll KNOW the world is ready to end...._grin_. The main thing is that together we'll see more than we can apart, even if there are disagreements. I think that's a pretty good gain in comparison to the effort.

-- Anonymous, February 25, 1999


CL, Perhaps you could outline what types of devices and systems you are testing and what the test procedures are?

I have pointed out to several colleagues that assumptions they had made about the system under test and the test procedure being followed was inadequate and flawed. Once the test procedure was corrected the units in question failed. And this was on products that had already been declared "Y2K Compliant".

Since I posted my message last night an additional 15,000 embedded systems with Y2K bugs have shipped worldwide.

--AJ

-- Anonymous, February 25, 1999


AJ, your post reminded me of being in a well-known department store last week, while my husband was picking up a computer connection. A lady asked him if he knew anything about computers because she wanted to buy the one on sale for a really good price. The first thing we told her was to ask if the computer was guaranteed to be Y2K compliant. This is the first person I've run across in weeks who had never heard of Y2K. Complete blank. She got a quick explanation, but when we left my husband commented to me (with a sigh) about the many computer and software sales brochures he sees with tremendous cut-rate prices -- and a whole lot of them he knows will not work properly ten months from now. Anyone reading, remember "Buyer Beware". Lots of people are incorrectly assuming that if it's new, it's Y2K good-to-go. Many businesses are making the same mistake. "Oh, our computers are only two years old so they're all right." Odds are, some of them aren't. Check! It's not only embedded controllers that may still be sold in a non-compliant state.

It also hasn't gotten much attention, but part of NERC's contingency planning recommendations is for utilities to pay careful attention to maintaining a "clean" site. To make sure they don't buy or integrate *anything* new into their systems unless it's been verified compliant first. A mistake in this area could throw a lot of remediation work down the drain.

-- Anonymous, February 25, 1999


Rick, Bonnie, and Mark: I enjoy your comments. It is true that responsibility is an issue that has yet to really see the light of day. I don't believe that it will be pretty when it happens.

-- Anonymous, February 25, 1999

The internet is an imperfect forum for this type of discussion, prone to mis-interpretation and the fallible communications skills of those who's communications skills may be muted by something as simple as poor typing. Let me correct some BAD inferences taken from my previous post - I'll assume that the error was mine.

1. I never called my predecessor incompetent. He was promoted to another department in a high profile position precisely BECAUSE of his competence. I meant to convey that MUCH progress has been made, intending to imply that proficiency is increasing.

2. Though it may be true, I did not state that utilities got a late start. Perhaps it is a fact, but not attributed to me.

NOW, let me address some of the 'points'.

1. If you do not know why my job would be in jeopardy simply for being here, I cannot help you. Speculate. That is easy and much practiced. Let the content of my posts give witness to my credibility. (oops, I'm in trouble now.) You are free to dismiss me and distrust me, discount, dispel, dispatch, disdain me.

2. I stated a fact. My utility does not count on co-generation to meet base load. They can all crash (they won't) and we will meet our load - no lights out. Ask your utility if they rely on co-gens - I'm not the one concerned, you are.

3. Ice will make trees fall on the lines of compliant and non- compliant utilities alike. If you survived the ice storm - good for you. I saw engineers abandon their pocket protectors to patrol lines and answer phones. Lineman packup and leave their families on Christmas eve. No Y2K involved. Man will never engineer a problem or a solution to compare to the scope of what nature can throw at you.

4. If you think government is the solution, I genuinely feel sorry for you. I posted once an opinion that some were advocating a gov't take over of the industy as a solution. This model has been tried. It's called Socialism. Try moving to a Soviet republic for the rollover, their government is calling the shots - they must be WAY more reliable.

5. The bit about the sick person and perishable medicine is a nice attempt to play at emotions. We deal with this every day we do field work. We deal with this every flood, hurricane, tornado, shoot even a car hitting a pole. We are sensitive to this already, its part of why we are so conservative in our engineering.

6. You don't trust 10% of your peers? I hope your not a pilot. With an attitude like that your would make a most excellent pointy haired one. Have you ever considered a career in utility (micro)management?

7. You say "If you will give me a list of applications you have tested, the independant company who verified your results, the methodologies you used, the list of suppliers you have, a copy of their independently verified upgrades, your status as to their implementation,..."

I'll tell you what - you post here the same information (verified test results showing failures) for each one of the devices that are gonna cause the calamities you stated, and I'll consider breaching my moral code and betraying the non-disclosure agreements I've signed.

More seriously, I use a modified version of the GM test plan. EVERY utility I've ever talked to uses the same. I listed (against my better judgement) the name of a relay vendor with test results and procedures on the web. Won't do that again. Ask Rick, see if he will. Ask your utility if they will provide you an inventory list so you can check out the web pages yourself.

FactFinder it really would be helpful if you could shed some light on Dr. F's embedded chip paper.

-- Anonymous, February 25, 1999


Bonnie,

I forgot to mention in the name of honest disclosure, I am a 2nd hand source on generation. Please keep in mind. (learned this from Drew)

-- Anonymous, February 25, 1999


CL, Back in November I gave two examples in this thread:

"Are power plants (and others) testing applications or the embedded chips themselves for y2k problems?" http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000G3V

I would be interested in your take on these two scenarios. Have you run across anything remotely like them in your test efforts? Do any of your PLCs use time/date?

What light would you like shed on Dr. F's paper? I don't think there is anything in it that I don't understand pretty well.

Regards, AJ

-- Anonymous, February 25, 1999


CL, My responses in brackets...

1. I never called my predecessor incompetent. He was promoted to another department in a high profile position precisely BECAUSE of his competence. I meant to convey that MUCH progress has been made, intending to imply that proficiency is increasing.

Your quote.... "Utilities got a slow start. When I assumed this role, my predecessors primary accomplishment was attending meetings." [You are right, you never did call him/her incompetent... I DID.. If the person was in the job for over 2 weeks and did nothing other than go to meetings then he/she was incompetent in my book. But then again I guess that would depend on what his/her secondary accomplishments were.]

2. Though it may be true, I did not state that utilities got a late start. Perhaps it is a fact, but not attributed to me. [You are right, I am sorry for that misquote, please forget my comments]

NOW, let me address some of the 'points'.

1. If you do not know why my job would be in jeopardy simply for being here, I cannot help you. Speculate. That is easy and much practiced. Let the content of my posts give witness to my credibility. (oops, I'm in trouble now.) You are free to dismiss me and distrust me, discount, dispel, dispatch, disdain me. [I still don't understand and I do...dismiss, etc..]

2. I stated a fact. My utility does not count on co-generation to meet base load. They can all crash (they won't) and we will meet our load - no lights out. Ask your utility if they rely on co-gens - I'm not the one concerned, you are. [Exactly my point, Thank you for agreeing with me that I should be concerned.... I am actually so concerned that I have started preparing for the worst and hoping for the best.] 3. Ice will make trees fall on the lines of compliant and non- compliant utilities alike. If you survived the ice storm - good for you. I saw engineers abandon their pocket protectors to patrol lines and answer phones. Lineman packup and leave their families on Christmas eve. No Y2K involved. Man will never engineer a problem or a solution to compare to the scope of what nature can throw at you. [Like I said in my original post we owe them a HUGE debt of gratitude... BUT will those people be there if their families are in danger from no food, no gas for their vehicles etc..] 4. If you think government is the solution, I genuinely feel sorry for you. I posted once an opinion that some were advocating a gov't take over of the industy as a solution. This model has been tried. It's called Socialism. Try moving to a Soviet republic for the rollover, their government is calling the shots - they must be WAY more reliable. [I have less confidence in the Federal Government than I do in your assurances. But like I said in a previous post, IF the Federal Government decides it needs to "take over" your power plant, they can and they will and if you must be killed for the "safety of others" then you will be killed.] 5. The bit about the sick person and perishable medicine is a nice attempt to play at emotions. We deal with this every day we do field work. We deal with this every flood, hurricane, tornado, shoot even a car hitting a pole. We are sensitive to this already, its part of why we are so conservative in our engineering. [No you don't deal with it everyday you do field work, there is always backup from someplace or rerouting power from someplace, there is a good possiblity that there might not BE any backup or extra power.] 6. You don't trust 10% of your peers? I hope your not a pilot. With an attitude like that your would make a most excellent pointy haired one. Have you ever considered a career in utility (micro)management? [What I said was I don't trust even 10% of my peers to do as good a job as I would, it only makes me good at what I do, just like you]

7. You say "If you will give me a list of applications you have tested, the independant company who verified your results, the methodologies you used, the list of suppliers you have, a copy of their independently verified upgrades, your status as to their implementation,..." I'll tell you what - you post here the same information (verified test results showing failures) for each one of the devices that are gonna cause the calamities you stated, and I'll consider breaching my moral code and betraying the non-disclosure agreements I've signed. More seriously, I use a modified version of the GM test plan. EVERY utility I've ever talked to uses the same. I listed (against my better judgement) the name of a relay vendor with test results and procedures on the web. Won't do that again. Ask Rick, see if he will. Ask your utility if they will provide you an inventory list so you can check out the web pages yourself. [I'll pass thanks... But it would be nice if you could address your contingency plans for fuel and spare parts but don't worry I am not holding my breath]

ANDREW, I wouldn't be waiting for a response they don't understand or take into account the big picture and understand the fragile nature of the supply chain and just in time inventory. Those are questions they will not answer. As long as they can say that their plant is fine, they think everything else in the world will be fine.

FactFinder it really would be helpful if you could shed some light on Dr. F's embedded chip paper.

[Answers on embedded chips are worthless, Bonnie tried to point that out but you changesd the subject... Why are we spending all this money, why are governments at all levels, creating contingency plans??? Oh that's right they are doing all this because there are NO problems, sorry I forgot.

-- Anonymous, February 25, 1999


Mark,

Thank you for that excellent response. It helps. I'm feeling better and better about the preparations I've made. (Probably not enough).

If I wanted to watch the "Artful Dodger" I would go to a play.

-- Anonymous, February 26, 1999


Story from Omaha Public Power District on Embedded Systems:
 
Brad Pence, manager of nuclear process computing services for Omaha Public Power District, worries about cases where the vendor claimed its product was "Y2K compliant" or "Y2K ready," but OPPD's testing revealed a failure.
 
[snip!]
 
Taking control of embedded systems
  Of OPPD's 2,300 Y2K-suspect systems, Pence says only about 10% are "priority-one show stoppers. These are the ones that would cause us to have to shut the plant down and stop producing electricity if they failed," he says. For those systems, in addition to contacting the vendor for compliance information, Pence's group is doing its own testing to determine the extent and type of the Y2K failure. Ominously, there have been cases where the vendor claimed its product was "Y2K compliant" or "Y2K ready" where OPPD's testing revealed a failure.
 
"We've had some vendors say everything's good to go, but we've found out it's not. It's a little scary," says Pence.
 
Link: http://www.datamation.com/PlugIn/issues/1998/september/09y2k.html
 
Dan
 


-- Anonymous, February 26, 1999

Per AJ's reference to Dr. Mark Frautschi embedded controls paper:

In an earlier message on this bboard, Dr. Frautschi pointed out that his paper is a "living" document, and he welcomes all input / criticisms. Again, the link is:

http://www.tmn.com/~frautsch/y2k2.html

-- Anonymous, February 26, 1999


AJ,

I've read the posts you reference, interesting but I do not think they apply. We have a very small installed base of plc's in my part of the system, most all on non-critical functions. We are testing from the top down. Picking the highest practical level of integration and testing that way. Mostly bench top with interconnected devices when applicable, and then discreet testing of individual devices. Chip level is impractical and I don't think as effective. There is at least one utility that has completed this type testing and is drilling down to test using a device that monitors RS-232 communications and monitors data for possible date transfers. I'm not.

We do not use pc based plc's. The environment in a substation is extremely harsh and coupled surges and transients wreak havoc (you may be familiar with IEEE Surge Withstand and Fast Transient standard?). I would question the ability of hardware to survive in the substation that doesn't pass this standard or it's European equivalent. Along the same lines, we do not use AC to power our plc's. Utility protective devices must be available at all times, even during loss of station service AC potential. We use huge batteries 125VDC or 48VDC to provide power supply voltage to our protective devices (as well as provide open/close potential to HV circuit breakers). None of our devices "reboot" during loss of AC, and if we were to lose DC then Y2K would be the least of our headaches.

All of our plc's are off the shelf compact versions from a major vendor. None of the models I use contain RTC chips. I do not use code to create a day of week or yearly calander function, though timers are used to "build" clocks to time functions at hourly.

Hope this answers your concerns? Perhaps you can comment on something I was told by an engineer from a major RTC manufacturer. I was concerned about a rumour I had heard - A microcomputer chip with an embedded RTC would shut down if the RTC wasn't Y2K compliant - even if the RTC was not wired out and totally unused. He said this was a myth.

Another question. Say a device has no date function and no MMI, no external comm port, and no battery to sustain an embedded clock when de-energized. I understand you to say this device could still fail by incrementing from a non-volatile "birth date". If these were to fail, wouldn't they be a random event that would appear to be a normal bathtub curve failure? We have equipment failures now - even on electro-mechanical devices. No big deal because they do not manifest simultaneously in a wide spread application. Is this really a problem? How would you test for this (maybe remove each chip and burn a new "pre-rollover" date and time?)??

Thanks for your reasonable post. It was worthy of a response.

-- Anonymous, February 26, 1999


Dan,

We obtain vendor certifications for all devices. Critical devices are all tested in house as well. We do not trust vendor OK's. We do trust vendor reports of failures. This assesment is good enough, we fix/replace and then test the fix. No need to check the air pressure in a flat tire before changing.

Also look at source of vendor opinion. Sales, marketing and legal types are dismissed in the same manner some posts here are dismissed as being of no value. We try to follow up to talk or correspond with an actual engineer AND obtain vendor test results.

Non-critical items same process. Testing of these will not be done unless ALL critical device testing, remediation, and all contingency plans are complete and effective.

-- Anonymous, February 26, 1999


Well done cl - I think you acquitted yourself well enough. I'm sorry you seem to have to shoulder burden of doubt and scepticism (to which I subscribe) almost alone. But I like to think with Bonnie's pointed questioning and reasoned arguments, together with Rick's commitment to get at the truth, you have many more allies than you know.

Just a thought, but am I right in noticing that (with some exceptions) we are no longer seeking the truth where we once thought it lay, i.e. somewhere between Toms Take (meltdown) and definite/serious/shortlived disruption. Does it now rest somewhere between potentially/serious/shortlived/ localised disruption to utilities (the majority on this forum, Koskinen, FEMA, the UN, myself) and nothing significant at all (cl/FactFinder, DeJager, most Governments in practice)?

If so, I'm feeling better already. My only problem is that I live in Europe, where we already have a blackout - an information blackout (notwithstanding the sterling efforts of Menno). Is the US still running those green card lotteries?

-- Anonymous, February 26, 1999


CL,

It sounds like the area you are remediating is well under control. It's a Good Thing(TM) that none of your PLCs are PC based and also that they are not programmed to use time/date.

It sounds like the electrical environment within the plant is very similar to what we do in the telecommunications business (48VDC battery backed systems, solid grounding, noise suppression, etc). However, there are plenty of "PC based" systems that you would never know are PC based and are used in these types of environments. One example of a embedded PC board that might reside inside something like a GE Fanuc Series 90(tm)Micro (http://www.ge.com/gemis/gefanuc/PLC-PC/90micro.html) resides here: http://www.ampro.com/products/coremod/cm-pc.htm (Of course GE Fanuc would probably not OEM from Ampro they'd make it themselves).

The fact that none of your ladder-logic uses day-of-the-week, year date or absolute time is also a Good Thing. Unfortunately your case does not really alleviate my concerns because it sounds like you are not using the types of equipment about which I am concerned. Perhaps this is good news for the Power Industry but I am not willing to conceed such a sweeping positive outlook based on a sample of one. Though I do thank you for your input.

Here are just a couple of examples of systems about which I have quite some concern: Ge Fanuc PC Control, based on Windows NT (ugh!) and no mention of Y2K (http://www.ge.com/gemis/gefanuc/PLC-PC/540000.html) and Allen-Bradley PLC-3's, PLC-5's and Pyramid Integrators -- at least Rockwell Automation / Allen-Bradley are being up-front and public with their Y2K disclosure (http://domino.automation.rockwell.com/webstuff/y2k.nsf/Pages/Brands-A llen-Bradley-Known+Issues?OpenDocument).

The fact that Ge Fanuc does not claim Y2K compliance on their web page and also force you to fill out a form to get Y2K information pretty much convinces me that they've got problems (if they were Y2K compliant they would be making a big hoopla about it).

A couple of months ago I did a quick survey on the web to see how many different PLC manufacturers I could find. Without trying very hard I found over 30 including: Allen-Bradley, Aromat, B&R Industrial Automation, Berthel gmbh, Control Microsystems, Control Technology Corporation, Cutler Hammer/IDT, Divelbiss, Elsag Bailey, Festo Cybernetic, Fuji Electric, GE-Fanuc, Grayhill, Honeywell, Idec, Keyence, Kirchner Soft, Klockner-Moeller, Microconsultants, Mitsubishi, Modicon/Gould, Moore Products, Omron, Opto22, PLC Direct/Koyo, Reliance, Siemens, Sigmatek, SoftPLC/Tele-Denken, Square D, Toshiba, Triangle Research, and Z-World. Most of whom have Y2K problems of one degree or another.

Thankyou for your feedback, by lurking in this forum I am learning a lot, and not just Y2K related either. I also hope that my occasional post is informative to others.

I'll comment on your questions in a separate post.

--AJ

-- Anonymous, February 26, 1999


CL, regarding your specific questions:

With regards to "A microcomputer chip with an embedded RTC would shut down if the RTC wasn't Y2K compliant - even if the RTC was not wired out and totally unused".

This is mostly myth but not entirely. First of all there are so many permutations of microcontrolers units (MCUs), microprocessor units (MPUs), realtime clocks (RTCs), and other embedded chips for peripheral support that I'm positive we could find at least one example of a design where the above statement was true. Furthermore many MCU/RTC combinations are not self evident on visual inspection. More and more products embed MCUs and RTCs into Application Specific Integrated Circuits (ASICs - a single black IC chip, see: www.vautomation.com/prod.html). Also of note these ASICs are not counted in the 4 billion number I quoted earlier. ASICs are counted as ASICs not as MCUs even if they contain an MCU. So, it's not impossible but not highly likely either.

On the other-hand, a much more common and real-world scenario is the IBM PC compatible embedded system and its asscoiated RTC. Prior to late 1997 / early 1998 almost all embedded PCs (like the Ampro CoreModules mentioned in my previous post) had the potential for Y2K failures. Between 100 and 200 million of these ship each year.

You said: "Say a device has no date function and no MMI, no external comm port, and no battery to sustain an embedded clock when de-energized. I understand you to say this device could still fail by incrementing from a non-volatile "birth date"."

To the best of my knowledge this one is completely myth. I know of no such device or failure scenario. But then again ya never know (it might be misconstrued example of an embedded PC failure mode).

--AJ

-- Anonymous, February 26, 1999


AJ,

I seem to recall you mentioning PBX problems. I have searched posts in vain for the example. I do not recall hearing of any PBX problems at EPRI embedded chip conference that would disrupt communications. Did you find such a problem? Do you attend EPRI or can you report any to EPRI for exchange with members at April conference? The only problems I recall were ones that impacted peripheral functions like call date/time/duration. I don't think we use these functions in our company owned PBX's. What can you tell me?

Thanks for your informative replies.

-- Anonymous, February 26, 1999


All of the electric utilities in Japan have reported that they have found nocases where a year component is used in the generation or distribution of power. Japan uses 40% nuclear, the balance in fossil with a small, man-made hydro capacity used for peak-load demand in the early afternoons. Most of the generating hardware is from U.S. or European manufacturers, thus technical support and information exchange is maintained with the manufacturers.

It seems reasonable that the same findings would be true in other locations. Westinghouse reactors plus GE and Siemens turbines are in more than just a few countries.

While this is excellent news, it must be tempered with the news that they are not portraying control, recording or maintenance systemsin the same light. Japan has the highest substation automation rate in the world. Hardware isbeing replaced and I know of no company that arbitrarily replaces hardware. The software installed to increase efficiency and remote data analysis is still being remediated.

Newspaper reporters read that no problems in power generation or distributionhave been found and write the storyline: "Electric utilities are OK on y2k." Control, communications and recording (not necessarily maintenance) systems do affect today's "power web." Therein lies the problem. Sometimes there are date components. Sometimes they are used. Sometimes they are used by different applications in different ways. Sometimes they are used in ways people are unaware of because they never had to think about it before.

A "great unknown" rollover failure rate of 0.001% starting "about" the first of next year and continuing for "awhile" is something everyone would like to be able to pin down a little tighter. We should have the results in a few years. The "greater unknown" of control, communications and recording within the industry isthe problem, not generation and distribution. We all know those elements are "OK."

-- Anonymous, February 26, 1999


All of the electric utilities in Japan have reported that they have found nocases where a year component is used in the generation or distribution of power. Japan uses 40% nuclear, the balance in fossil with a small, man-made hydro capacity used for peak-load demand in the early afternoons. Most of the generating hardware is from U.S. or European manufacturers, thus technical support and information exchange is maintained with the manufacturers.

It seems reasonable that the same findings would be true in other locations. Westinghouse reactors plus GE and Siemens turbines are in more than just a few countries.

While this is excellent news, it must be tempered with the news that they are not portraying control, recording or maintenance systemsin the same light. Japan has the highest substation automation rate in the world. Hardware isbeing replaced and I know of no company that arbitrarily replaces hardware. The software installed to increase efficiency and remote data analysis is still being remediated.

Newspaper reporters read that no problems in power generation or distributionhave been found and write the storyline: "Electric utilities are OK on y2k." Control, communications and recording (not necessarily maintenance) systems do affect today's "power web." Therein lies the problem. Sometimes there are date components. Sometimes they are used. Sometimes they are used by different applications in different ways. Sometimes they are used in ways people are unaware of because they never had to think about it before.

A "great unknown" rollover failure rate of 0.001% starting "about" the first of next year and continuing for "awhile" is something everyone would like to be able to pin down a little tighter. We should have the results in a few years. The "greater unknown" of control, communications and recording within the industry isthe problem, not generation and distribution. We all know those elements are "OK."

From Japan...

-- Anonymous, February 26, 1999


Sorry about the double vision.

-- Anonymous, February 26, 1999

CL,

Here's one reference: http://www.techweb.com/se/directlink.cgi?IWK19970602S0065

For more just go to "www.altavista.com" click on "Advanced" in the "Boolean expression" box enter "pbx near y2k", and click on "Search".

--aj

-- Anonymous, March 01, 1999


Moderation questions? read
the FAQ