xBob, in a current thread said:

"However, I am very disturbed by the lack of continued emphasis by you and all the other engineers on the insignificant size of the problem to begin with..."

This is amazing. I don't have time to search through the volumes of posts by others, but have found some by searching on my various threads. Read them all, and see if you can determine EXACTLY WHAT could have been done to empahsise the scope of the embedded chip problem in utilities:

From June 16, 1999 the following (Note the first statement):

1. No mission critical Y2K devices failed in a manner that would in any way jeopardize power transmission and distribution. These results I have in my posession and have seen and heard from countless utilities that have independently tested with the same results. 2. We can operate the system manually. We do it to a large extent now, as so ably noted by Dan, Engineer etc. We have drilled operating with a failure of an external provider (telecom) that would defeat SCADA. The drill proved that this can be done. We are ready to operate this way. Name the system that will failed (based on testing) to cause us have to operate this way. Describe the sequence of events that will lead to you cranking your high dollar toys. This challenge has been put forth before. Describe your scenario and name the faulty devices and their failure modes. 3. You misinterpret the analogy directed to Gordon. The difficulty of flying a plane without a trim wheel relative to using a trim wheel - the difficulty of landing an airplane with manual gear vs. motorized doesn't indicate the level of competence necessary to fly - just the relative difficulty of some contingencies. Not that I'm knocking the abilities or contributions of all MSU grads at the McDonalds around the country. (grin - GO IRISH)

-- cl (cl@sky.com), June 16, 1999.

NEXT - a call for Rick to honor his word and disassemble this site based upon an authoritative statement from NERC:

All Good Things Must Come to An End greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

Rick, In my rather controversial appearance on this forum many months ago, we had a rather interesting exchange. Unless you had your "Irish" up and emotion rather than intellect was doing your speaking - you have a promise to fulfill. In the thread http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000REi you said (clip follows) Second, when will your company be saying they are "Y2k Ready"? Or compliant? Or whatever the heck the buzzword is today - basically a reassurance to the customer that the chances of power failure on 01/01/2000 are no different than today. If you can answer this question, there's no need to read further. Actually, if every individual power company would answer this question, there would be no need for the euy2k.com website, and I could shut down the website and move on to greener pastures. There's a lot more $$ in SAP implementations than there is in Y2k. More of a future, too. (end clip) Now, read a clip from the article posted in another thread by Art quoting the head of NERC. (begin clip)In a status report Friday, the council said it appears 2000 will have minimal impact on North American electric systems. Sufficient generating capacity should be available to meet the nation's needs, and transmission outages should be minimal. At the end of March, three-fourths of all the testing and repairs that are needed had been completed, compared with 44 percent in November. "The bottom line," said council President Michael R. Gent, "is that for the typical person or business in North America, the supply of electricity will be like that on any other New Year's Day." (end clip) This is what you requested. The EPRI conference attendees had no show stoppers that would contradict the NERC statement. The EPRI conference corroborates the NERC conclusions. These are not corporate spinmeisters at EPRI, they are the folks in the trenches doing the tests - your former comrades in arms. Good luck with your SAP implementations, I sincerely hope you will prosper. Thanks for keeping this site up for as long as it was needed. It was indeed an interesting exercise observing engineers trying to explain an engineering solution to a public of "normal" folks. I guess Dogbert is right, engineers need to do some more socializing.

-- cl (cl@sky.com), May 05, 1999

AND,,, From March of 1996. Please note item #6. (This is also an interesting retrospect on the difficulty I had rationalizing the expense of contingency planning given the solid test results).

Jon, 1. The Seabrook audit thread did not prove anyone was lying. It just proved that we are unable to interpret it correctly. There were several possible truths behind the words as they were constructed. You chose the sinister interpretation, I chose the innocent. Neither of us can claim to represent the objective truth. 2. Home based Y2K test for you. 1. Go to any hardware or electronics and buy an inexpensive voltmeter. 2. Make sure it is on a scale larger than 120VAC (do not set it on continuity test or ohms scale). 3. Very carefully insert the probes into a wall socket. 4. Unplug all the phones in your house. 5. Observe the voltmeter over several days. (If you want to go all out get an expensive voltmeter that also measures frequency and look for deviations from 60 hz) It has been discussed here how the system will continue to operate in the absence of external communicaitons. Shoot, it has even been explained that the system will stay up even in the absence of internal SCADA communications. This is fertilizer. 3. Money spent does not indicate progress or success (unless you are a fan of big government and look to Uncle Sam to care for you like an adopted orphan). EXAMPLE (disclaimer; do not bite down while tongue is firmly planted in cheek): One big portion of the supply chain has been overlooked. All fossil fuel plant, in addition to coal/oil need oxygen. My company has a 3 million dollar budget to ivestigate whether the phosysthesis process is Y2K compliant. Thats right, no one is looking into photosynthesis and the cascading impact of a Y2k failure on the oxygen supply. We have spent none of this money. Does this imply a failure of the photosynthesis process? NO. EXAMPLE 2: A friend recently had gutter, siding, and roof damage due to a horrible gas explosion in a nearby pizza business. His insurance company estimated the amount of damage and the cost to have it repaired using a contractor. Being a handyman, and with contractor friends to help him get materials at cost, he has only spent a fraction of the money estimated (30-40%) and disbursed. Is he complete? Is his work adequate? Can you tell by the percent spent? 4. Your absolutely right about the double standard. Unfortunately it cuts both ways. I feel incredible conflicted in jumping through hoops for contingency plans for failures I have proven through test will not happen. Why waste the money? Because the combination of lawyers limiting our releases over concern for litigation and people like you who always see the sinister leave the utilities no acceptable choice. If we do not make contingency plans  the sky is falling, they arent preparing. If we make contingency plans, the sky is falling  see, the utilities are preparing because they know its gonna happen. You gotta leave room for one option of a binary choice to be correct. 5. It doesnt matter what utility folks say or do here, no opinions will be impacted  you will believe what you came here believing because that is what you want to believe. This site has been a culture unto itself, and to the extent it calms the fears of those whose opinions are set to doom, then it is a good thing. Fact Finder, Y2kGuru, and others might as well pack it in and get on with remediation and contingency planning, this effort at influencing this culture by shining light from the inside is futile, and a waste of our collective time (in my humble opinion). 6. So I will take my leave from these discussions at above 90% complete in mission critical embedded systems complete, still having found no problems beyond the cosmetic and confident that by the time I complete the critical items in 2 weeks I still will have found nothing of significance. Thanks to all who have engaged in reasoned (sometimes) discourse here. I have found that participating in this culture has been both challenging and enjoyable despite the frustration level. I pray all will be well, and encourage to begin to prepare spiritually for Jubilee 2000 with the same vigor you are preparing for the temporal Year 2000. May God bless each of you.

-- chicken little (cl@skyisfalling.com), March 30, 1999.

FINALLY, this exchange showed up on a thread that came up in my search. It certainly shows that Art had a clue, even if he didn't persist in posting.

Linda, No one can ever prove to you that something will not happen. It is quite impossible to prove a negative, and if I can't prove it to myself, I would never promise you. I can tell you that I identified over 1000 devices, and about 100 of these are used in a way that they would cause an outage if they failed. I have tested about 75% so far, and found not a single significant problem. I have been to EPRI and not heard of anyone else in my area of specialty report anything different. I have knowledge of only my area. I cannot predict others area or outside dependencies. Things look good in my area, and my friends in generation say similar, but they have totally replaced (updated) an important system. Diligent utilities will find such systems and fix and replace. Don't know the impact if this system had been ignored. Didn't help you did it?

-- art (art&science@protrelay.com), February 16, 1999.

Good response, art. My point is, no one can make absolute statements about y2k, no one. There are just too many variables. So, I don't think it's particularly helpful to rant at either the optimists or the pessimists. I think it's important that they each debate with all the information at their disposal. That gives us, "the ignorant" a better opportunity to make out decisions. I am still expecting more information before I am sure which side I will come down on; however, I realize there will never be the "ultimate" statement of fact as to what will happen. But I want to hear it all, the good, the bad, and the ugly. Why not hear both sides and allow both sides to speak? Seems there is fear on both sides.

-- LindaO. (urloved@hotmail.com), February 16, 1999.

Linda, Yes, I did speak in absolutes. None of the types of devices that I tested will fail. Right now that is 75% of the devices from the outside of the gen station to the customer pole. I absolutely will finish the other 25% months ahead of the NERC deadline. My peers at other utilities have tested most types of the 25% I have not gotten to. After reviewing their procedures and results via EPRI, I state absoloutely that the devices they have tested will not fail on their system or mine. Hope this helps. And yes, I have guaranteed this to my immediate manager. I will be held accountable.

-- art (art&science@protrelay.com), February 16, 1999.

xBob, no one has been on this board with "I told ya so"'s. You almost seem to invite it?

-- Anonymous, January 13, 2000

Answers

Rick,

Can you format the crap I just posted?? Sorry, still html challenged.

-- Anonymous, January 13, 2000

You are right CL,

"xBob, no one has been on this board with "I told ya so"'s. You almost seem to invite it?

-- cl@sky.com (cl_sky@excite.com), January 13, 2000 "

Yes, I would like a quantifiable, "I told you so.", rather than vague generalities we generally got.

In your search you have come up with one 'I told you so, almost'. - your first one where they have identified 1000 units, cleared 900, and tested and cleared 75. That translates into 99.75% of the units will be no problem, but we still have 25 percent of our potential failure points to clear (note this is just 2 weeks before the NERC y2k ready declarations of 1 July 99. Actually, this is pretty great, and indicates a minisule problem. Did you ever find, we tested 1000 systems and all 1000 were OK, or something similar?

Balance this with the spin - 'were spending billions' spin, eastern and communist block countries will crash, the mid-east is 18 months behind and we only have 6 months to go, 'we don't have no facts but it ain't no problem' spin, the 'i didn't have sex ...' presidential spin of don't worry be happy, and finally the NERC edict (it's OK to lie and we will accept lies and cover up for you) spin.

What you get is a huge percentage of, 'it's not a problem', spin, based on very few quanitifable facts, and actions which were totally in opposite of the spin being promoted.

Yes, I want some 'I told you so's', where there were some quantifiable facts (which engineers deal in).

Question - would you rather buy a used car from a salesman who says, 'Yeh, it runs great.' or after a mechanic says - 'I tested the compression, suspension, fuel and electronics systems, and it seems to be good.'?

Personally, I feel that the world bought a used car from the salesman, lucked out and got a great deal, and the salesman is now more amazed than the customer.

xBob

-- Anonymous, January 13, 2000

Happy post-y2k, xBob. Yes, there are several utilities that reported that NONE of the embedded systems tested failed. One example you can find on the web is

www.idahopower.com/y2k/faq.html#a10

"We found no date related failures that would stop performing its basic function (and potentially interrupt service."

There are several others, if you would like them.

Furthermore, take a gander at Gerry Cauley's NERC presentation slides (they are in power point form) from the August meeting in Chicago. The first slide says something like "1 to 3 percent failure rates are being reported in embedded systems, and only a small percentage of those were more than cosmetic".

Thus the second item shows that the problems found were very, very small in number, and this was based on thousands of tests performed throughout the industry.

-- Anonymous, January 14, 2000

Dan,

I appreciated your studied responses throughout, yet once again, no failures is not of great meaning. If you have 1 system, 1 failure is totally catastrophic. If you have 10,000 systems, 1 failure is minisule. Once again, "we had no failures" places no magnitude on the problem, and that is the point I am trying to get at.

Scattered reports, here and there of no failures is even more misleading. Please tell me how many of the 6800 power plants, or what ever the correct number is, had no failures? How many of these reported that last summer? How many systems were affected, how many problems were found.

If we had an energy department with any guts, instead of delegating the whole show to a coverup - NERC, dedicated to assuring that there was no good knowlege available - the DOE chickened out - pre-cut and wrote all the press responses, and lucked out that their own propaganda turned out to be true.

As far as I can determine, there has been no coherent report on the magnitude of the problem, even yet. The best one is that all the systems, fixed or unfixed are mostly still here, so the problem was non-existant from the beginning.

My take on the whole y2k is that there was no y2k problem of any magnitude worth worrying about, and that the people who knew failed to organize, quantify or report this and caused thinking people, like those on this forum to be unduely concerned. It was one giant waste of time and there were no leaders smart enough, with courage enough, or organized enough to say so.

Don't you think 6 months (1 Jul - 1 Jan) is enough time to say - ' we have ___ power companies which have tested everything and we have fixed ____ problems, and have determined that ____ systems were non problems, and it works. After all it was said 'we're y2k ok, sort of', July 1. My big beef is that the news was great, and they covered up the good news.

My take is that y2k was not a hoax, it was just one vast exercise in stupidity and cowardness and propaganda, and we all lucked out, even though many of you guys busted your butts. I am not blaming you. I am blaming our leadership.

xBob

-- Anonymous, January 14, 2000

Dan,

One additional item - your note -

""1 to 3 percent failure rates are being reported in embedded systems, and only a small percentage of those were more than cosmetic".

From what I have seen, the actual failure rate in embedded systems was perhaps about 2/100ths of a percent, rather than the generally reported failure rate of 1-5 percent which would have been catastrophic.

-- Anonymous, January 14, 2000

To all,

I guess on of the things that really burned me the most, after all the work I and many of my friends did was our local news carried a story which ran something like -

"XXXX news has been notified that our local Entergy power plant will crash on y2k and here we have ______ the plant manager to discuss that." This was at 5:10 PM, 31 Dec, 10 minutes after GMT rollover.

Then they had the local Entergy power company manager come on in an interview and say essentially - No way - we rolled over last October and ran our whole plant on post 2000 dates and it worked fine.

They just conveniently forgot to tell us local people, in at least 4 interviews between October and December, where I watched power company management duck and dodge questions, with platitude that all was OK, rather than making that simple statement they came out with bull crap like all the propagandists - we're y2k OK.

Something similar went on in Florida where my brother lives, but they finally came out with it 2 days before roll over, that they had rolled their plant over in August and it worked fine. They just didn't tell anyone until 29 December.

This is what really burns me.

-- Anonymous, January 14, 2000

xBob,

I agree, and regret that official statements were contradictory often mixing prudent prep with everything is ok.

The rub comes in when the "prudent prep" part was not adequately identified as independent of, and unrelated to Y2K (ice storm for example) AND the everything is OK was couched in the non-committal language dictated by the lawyers.

This situation is what pricked my conscience into participating here. There was PLENTY of evidence available here and elsewhere on the net to conclude that utilities were ok. The search facilities here are weak, but I remember many discussions of no problems being found, the conclusive evidence of NERC report being dismissed as "lies and damn lies" etc. Shoot, I even remember someone here stating that they had not only completed their CRITICAL equipment, but also their NON-CRITICAL stuff (I remember congratulating them and being ridiculed by Lane as issuing a "press release").

It was all available. It was all here. There can be little doubt that utility folks here were confident of no problems, and stated that unequivocally. I had hoped that the other util types would search and find their threads from the past and post them here. The problem that kept folks from recognizing the truths being related here by those involved in testing, was the ambiguity of "official" press releases and a propensity to mistrust self-reported data.

-- Anonymous, January 14, 2000

"My take is that y2k was not a hoax, it was just one vast exercise in stupidity and cowardness and propaganda, and we all lucked out, even though many of you guys busted your butts. I am not blaming you. I am blaming our leadership."

xBob Have to disagree. As one who badgered my utility, especially on nuclear backup generator problems, and just got a response yesterday from the head guy that my questions made them really work harder to make sure things were all right, I'm proud of my role. But I am beginning to entertain the possibility that Y2k was indeed a "hoax" so that the population would prepare for disasters that may come with global warming while the "experts" can still disagree that warming is even happening. I, too, blame our spineless leaders and their lackeys in the press.

-- Anonymous, January 14, 2000

xBob, Well maybe those of us inside the industry didn't supply the facts and figures adequately, but we sure as heck tried to, please give some credit for the attempt at least.

I did a search, and found many posts that had numbers, percentages, and statements that we weren't finding significant problems. I will not link to these posts here, I have no desire or need to do an "I told you so" post. If you want to know what was stated by those in the industry, do a search for cl@sky.com, Malcom Taylor, etc.

For the record, I found much more very clear info and facts posted by cl than provided above, but do the search, and see for yourself.

I did find a couple of the more humoruous threads concerning the infamous Q7 transistor scare. The orignal thread was unfortunately deleted, and I don't think I saved a copy. If I were to do a link of some of my favorite threads here, that would be one :)

Regards,

-- Anonymous, January 15, 2000

Factfinder, Thanks for admitting that the industry didn't let the facts be known. I realize you did try very hard to get out what you know. Thankyou very much. I have had no doubts about your honest and integrity, and when you said your plant would run, I believed you. When the leaders said - no problem were ok, I didn't because they purposely obfuscated what facts were known.

Unfortunately, the leaders of industry and government tried their best and succeded in converting good engineering and statistical facts into legalistic gobledegook pablum and provided no backups for their grandeloquent statements of we're y2k OK, even when they were available. They lied and covered up the good stuff. It came out like " I didn't have ________, with that problem, y2k"

You guys in the trenches were probably muzzled and spoke mostly all in generalities or in things too specific to have great usful meaning in the question of - how great is the risk.

Y2k didn't happen, not because so much was fixed, but because so much didn't need fixed. Gary North has a recent article on his site which says there have been a total of 67 significant y2k failures around the world. 67, out of all the millions and millions of systems out there.

I agree that hundreds and probably thousands of problems were prevented by y2k fixes. But a few thousand out of millions and millions is still a fraction of a percent.

Here's to our lying leaders.

-- Anonymous, January 16, 2000

As one of Rick's "civilians" who has lurked around the edges, I would like to point out that industry, big business and .gov had a social responsibility to prepare a very long time ago, at least a year or so. At some point all these critical players decided their roll was merely to be ready in time for the rollover. It was very clear to me when I first woke up to y2k possibilities over a year ago that it would not be possible for very many people to truly prepare, and that market forces required that at least some get on with it. So, with the news as of the fall of 1998, NERC not knowing what to expect and unwilling to discount common mode failures at the time, and whatever other major unknowns were floating about, I had to take action to protect my family. It was inexcusable to me to wait for the luxury of adequate informationon how the utility industry was progessing.

In my mind what was truly needed was for a qualified team to have looked at what was needed, a very long time ago, make that information excruciatingly known so that utilities could prepare most efficiently (their costs *will* be passed on to the consumer), and their customers (whether individuals, small business, government agencies) would have known more realistically what to expect. As others have said, an appalling lack of leadership. I am underwhelmed by the circumstances that the utility industry "lucked out".

My greatest appreciation on this forum goes to those who were not only exceedingly knowledgeable but also willing to share that information in an exceptionally polite fashion. The generally condescending attitude of certain anonymous posters did little to assuage my concerns compared to the contributions of folks like Rick and Malcolm (and Robert Cook on TB2K). I appreciate the time all of you spent, it's just that some could have been far more convincing for the effort they expended.

Thanks in largest part to this forum, my assessment going into the rollover was that I would have power. (That was not at all my assumption a year ago.) However, in this age of deregulation, it was irrelevant to me what my utility company (which sends me the bills but generates nothing) had to say about its level of readiness. Terrorist attacks were all too possible. (A three day outage of our spy satellite system at one of the most vulnerable times we have ever had has more than justified to me the preparations I made). I also don't think the oil story is finished yet. And I don't think our leaders have a clue how they let us down on this one, it can only be repeated.

-- Anonymous, January 17, 2000

Bravo Brooks

-- Anonymous, January 18, 2000

very well said brooks ! thx brent

-- Anonymous, January 19, 2000