Case Failures

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

Don't know if this is old news, but IEEE has within its website a case book on documented Y2K failures found during testing. The case descriptions are detailed, documenting the problem, how it was discovered, impact on system and possible impact on business. It is fascinating (macbre?) reading. The failures range from the trivial to the potentially very dangerous. While not directly related to the electrical industry some examples appear relevant, (SCADA, HVAC, Building managment etc).

To my thinking, these examples documented by IEEE are as worrying as poor compliance percentages or concern with missed deadlines; they are more concrete. By this I mean I am uncertain what it means that an e-mail server is 85% remediated, does that mean I don't get junk mail in 6 months? But when the Institute of Electrical Engineers states these systems where found to fail when tested for century rollover and the failure in some instances would shut down the systems, I take notice. It may be difficult, with such a widespread problem that is not yet entirely fixed, to get accurate information on compliance status in the little time remaining. However, can anyone speak to the prevelance of any of these systems or embedded chips? Or for that matter, can anyone tell me how many of the non-compliant chips documented on Motorola's website were manufactured? It would be encouraging to hear that no one uses these chips anymore, or no one in the electrical industry uses them or if everyone uses them and someone knows this, we need to know. I sincerely hope Tesla isn't rolling over in his grave.

http://www.iee.org.uk/2000risk/Casebook/eg_index.htm#TopOfPage

Paul

-- Anonymous, June 20, 1999

Answers

Thank you very much Paul. I was to their site last year and they didn't have htis. IEE is a highly respected engineering group. Some of the findings are truely disturbing. Just to run the gauntlet to get such negative information posted on this site is quite an accomplishment, or even just getting your boss to listen. And this is just what they have found and reported, only a tiny tip of the iceberg. The petro-chem plant, over-pressurizing, is nice words for "BLOW UP". The high tech welding machines which won't track - and the mfg didn't know - tremendous problems for mfg world wide are just two examples. The flare stack which won't work. This confirms it for me for sure. Hills, here I come.

-- Anonymous, June 20, 1999

Just to make sure we don't mix apples and oranges...

IEEE is the U.S. based Institute of Electrical and Electronics Engineers.

IEE is the U.K. based Institution of Electrical Engineers.

IEE has been at the forefront of Y2k and particularly the embedded controls issues since at least 1996. Their website, research, and technical book published on the subject in 1997 are the seminal works on the matter.

IEEE was very late to the game (early to mid-1998). I think Dick Mills' latest column even addresses IEEE, and expresses his personal frustration with getting IEEE leadership to acknowledge that they had a role in research or developing strategies/solutions for embedded controls.

Even today, it appears as if IEEE has not really assumed a visible and public leadership role in dealing with Y2k. You can view the IEEE Y2k website at:

http://www.ieeeusa.org/Y2K/

Compare this against IEE's Y2k website referenced above. You'll see what I mean.

-- Anonymous, June 20, 1999


I was about to clarify IEEE vs. IEE, but you beat me to it Rick. As far as Y2K, IEEE wasn't even in the ballgame, a disappointment to me as a member. IEE has done a better, but not very much better, job. IEE repeats many of the common fallacies of the run-of-the-mill Y2K sites and embedded systems white papers. The embedded system failure reports are worth a read though, just to get a flavor of y2k bugs in embedded systems.

I urge others to take a critical and objective look at the failures reported in the IEE index I want to caution however, that from reading over these reports, a number of the "potential" catastrophic failure reports are similarly written, and may perhaps the work of an overzealous company hyping it's Y2K work, or perhaps confusion by this company as to how to interpret "Consequences of failure to the Business" part of the survey (i.e., does this refer to consequences of the Y2K failure, or of a system failure from any cause?). Examples: EG-54 appears to be a typical minor date problem, yet the "Consequences of failure to the Business" are written as though the equipment might fail. EG-52 says in one place will continue to work properly through millennium unless powered down and restarted in which case the date will be wrong. Consequences are "system stops"??? "Potential failure of air conditioning/ heating system, security systems etc."?? This report isn't even consistent with itself. My favorite here is EG-67 - the ultimate in hype!

In comparison, EG-49 is a well written and brief report that clearly addresses the consequences of the minor Y2K bug, not of a "system" failure due to "any cause". EG-47 may also be credible since actual testing was performed, assuming that the testing was proper (it often is not, and testing methodology can induce artificial failure mechanisms). A number of other reports here also appear credible since details are supplied in some cases. But without a listed source, equipment model information and the like, I would hesitate to use this information for anything other than getting familiar with the types of y2k bugs in embedded systems. You may even want to exercise the same healthy skepticism you use when reading information provided by utility industry insiders ;)

Regards,

-- Anonymous, June 20, 1999


Good clarification on difference between IEE and IEEE. Have worked with British and American engineers for more than 20 years who have been members of these societies. My brother is currently a member of IEEE. Agree, IEEE has been more lax on the issue.

Personally, I considered the IEEE reference rather than IEE as a typo, as the link address plainly showed IEE. They are both well respected organizations.

-- Anonymous, June 20, 1999


assuming that the testing was proper (it often is not, and testing methodology can induce artificial failure mechanisms)

That is no doubt true. Isn't the opposite true, too? Improper testing methodoloy can incorrectly assume success.

Been on both sides of that issue....

-- Anonymous, June 21, 1999



By the way, thanks for the post, Paul.

But, you seem to be forgetting something: information from people who hide behind pseudonyms is always to be preferred to information from groups like "The Institution of Electrical Engineers". Geesh.

Oh... no... wait... trusting pseudonymous posters is how we get all the wrong information... right?....

-- Anonymous, June 21, 1999


My apologies on the typos, I realized after posting that I had made an error, mea culpa, mea culpa. I had read at some time the distinction between IEEE and IEE and this may have contributed to my poor editing. I wasn't fully aware of these two groups differing Y2K philosophies, thanks for the clarification. If data is important enough to post it should be documented correctly, again my apologies. That said, this is the first reference I have where engineers document known failures from testing. I have no reason (eh, other then Y2K itself) to a prioir question the judgement or veracity of the engineers who contributed to these examples. So I am left with the question of how significant or severe a scenario do these examples point towards. Thanks again for the clarification. -Paul

-- Anonymous, June 21, 1999

Moderation questions? read the FAQ