Dick Mill's 1997 comments on Yourdon's Fallback (TB2000) Book.greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread |
POST:From: sams@brigadoon.com To: year-two-thousan-@egroups.com Subject: Dick Mill's 1997 comments on Yourdon's Fallback (TB2000) Book
http://www.albany.net/~dmills/fallback/fintro.htm
The title has changed to Time Bomb 2000 and chapter 5 became chapter 3.
I'm primarily motivated to comment on Chapter 5, about electric utilities, because that is my personal area of expertise. Allow me to introduce myself.
I've been a software engineer, working in power generation, and power grid control, and simulation for 29 years. I have personally contributed to the invention of the regulating systems for generation plants and for regulation of the grids. I've also had long and deep experience in simulating the way that power plants and power networks actually work. I've also have more than passing acquaintance with all aspects of how electric utilities work via so many years rubbing elbows with my clients. I've also done considerable work in the field of probabilistic risk assessment, so I've studied how things fail, and why, and how often. I guess that given all that relevant background makes it almost mandatory that I comment on Fallback's Chapter 5. You can find my resume at http://www.albany.net/~dmills/rjm.htm
*********
http://www.albany.net/~dmills/fallback/chapt5.htm
This page is part of Dick's comments on Fallback.(Timebomb 2000)
Power Control Principles
The book makes one serious misstatement in Chapter 5.
".. the computer software for electrical generating units has been written more carefully, and tested more thoroughly, than the business software in most companies. But these systems do have date calculations embedded within them (e.g., to regulate electrical generation or distribution in accordance with traditional hourly, daily, or seasonal variations in demand)"
The misstatement is important because it implies that date calculations are at the core of the principles of operation. Such is not the case.
The generation of electricity is regulated according to actual demand. There is no practical way to store massive amounts of electric energy, so it must be generated at the instant of demand. When you flip on the light switch in your bedroom, you actually cause the nation's electric system to increase the generation just enough to power the lights. The details are too technical to discuss here, but if you think about it, if the electricity can't be made in advance and stored, there's simply no other way.
The point is, the calculations for regulating power generators and power systems are in-general not date sensitive. Indeed, the regulating computer programs dont even need access to the date for any purpose. Hourly, daily or seasonal variations in demand have nothing to do with it. That's terribly important.
snip
Regulation Versus Protection
A second serious misconception in Fallback, is that it seems to confuse regulation of equipment with protection of equipment. Chapter 5 says,
"While its unpleasant to think about the power outages that could be caused by the computers shutting down, things could be even worse: an aberrant computer could, in theory, overload and burn out the generating hardware, or cause other irreparable damage."
Actually, regulating devices and protection devices are highly separated. This is true in all process control systems, not just electricity.
For example, a generator, or a power transmission line has a maximum current it can handle without damage. It is protected against this by a device which measures the current and opens the circuit breaker if the current is too high. This calculation is totally non date sensitive. It can not be overridden by any orders from an aberrant central computer. In many cases, the protective devices are not even computers, they are electromechanical relays.
In recent years, electromechanical relays are being replaced by microprocessor based solid state equivalents, called intelligent electronic devices, or IEDs for short. IEDs, take on the added functions of communicating to the central computer, and to create historical archive records. These functions might be date sensitive. However, the basic protective functions are still not date sensitive, and there is no reason to assume Y2K vulnerabilities. Further, each protective device tends to be free standing, with its own computer in its own box. Further still, the different devices overlap in function. Thus if the overcurrent protection fails, the protected device may overheat, and a completely separate overtemperature device will provide backup protection. Even if the central computer goes belly up, these independent boxes keep on doing their thing.
snip
The reason I'm explaining this here is that Fallback's premise that Y2K caused problems might damage generators and other equipment, thus resulting in month or longer outages, is not a reasonable extrapolation. In my opinion, Fallback's statement,
"The most likely scenario, in our opinion, is the blackout that lasts for a couple days; a less likely scenario, but one we feel should not be ignored, is the one-month blackout. Why? Because it could take that long to fix whatever Y2000 problems are discovered in the hours after midnight on December 31, 1999; and it could take that long to restart the system.",
is unfounded. It shows ignorance of protection versus regulating functions.
Robust Systems
The book makes the point.
" How Could Such Failures Happen? The first thing to realize is that such failures already have happened, on numerous occasions the only difference is that they werent caused by Y2000 bugs" That isn't the only difference. There has never been a widespread month long blackout in more than 100 years of experience. I think the statement also implies that the less than perfect history, makes Y2K extrapolations more plausible. Actually, just the opposite is true. It would serve the public better to point out that it is the system which has never failed and thus which remains largely untested that is most vulnerable to complete collapse. The more frequently a system fails the more real life experience we have about how they fail and the consequences of failure.
The non-technical reader reading the non-technical descriptions of the ripple effect and descriptions of the power system might reasonably come away with the impression that the systems are fragile. In other words, they might believe that nearly everything has to work right, or else it blacks out. The reality is just the opposite. Steam power generating plants are enormously complex, the power system itself is enormously complex, and dependent on all these power plants. The power system has operated for more than 100 years, and all during that time there has never been an instant when there were not thousands of malfunctioning, or aberrant devices participating. It's also probably true, that there has never been an instant when some isolated customer wasn't inadvertently without power. There has never been a case, hurricanes and tornadoes notwithstanding, of a national blackout, or a regional blackout lasting much more than 24 yours. That knowledge puts quite a different light on things.
snip
I challenge anyone to come up with a substantial calculation that shows that the morning of January 1, 2000 will stress the power grid even as much as a typical ice storm.
http://www.albany.net/~dmills/fallback/murphys.htm
Murphy's Law Doesn't Apply Here
I recall hearing a statement regarding the Apollo moon mission launches. It is true that a single loose wire, or a weak rubber O ring can cause disaster. On the other hand, the Apollo mission was so complex, that if every component was required to work correctly for the mission to succeed, then only one in five million launches would have succeeded! It is misleading to emphasize the vulnerability to simple failures, while failing to point out the overall robustness. This logical flaw seems common in the way many commentators on the Y2K problem have been approaching the problem. Allow me to pick on a typical example from Fallback Chapter 13.
There is a federal (US gov) standard for a cars emissions control systems logging any failure conditions of the components, for example a fuel injector being open longer than it is supposed to due to some dirt in the fuel. The name for the logging system is OBD2 (On Board Diagnostics 2).
This next is informed speculation. Suppose there is date-time logging for the failure. Suppose that the date routine for some of the software is (surprise) not Y2K compliant. Suppose the failure mode is either to lockup or refuse to run the car (unlikely but not impossible). I have seen statements in both directions - that some automotive engine control processors will or will not fail after The Day. It seems to me that it is likely that those who know (because they wrote the software) are probably contractually bound to keep quiet. The rest of us are just guessing.
The supposition is that such a failure has a serious effect on the primary function of the car. The credibility is buttressed by the suggestion that corporate types are engaging in a conspiracy of silence. In many cases, (not this one), it ends with an appeal to you to spread the word so that someone in authority might do something about it.
The first thing that should strike you about this example is that it precisely follows the recipe for creating a successful urban legend. I don't want to be pejorative, but I do want to point out this as one reason why the whole Y2K problem has trouble gaining credibility in some circles. Our anecdotal examples, are very hard to distinguish from urban legends.
The logical flaw in the above example, and in many others, is that they implicitly invoke Murphy's Law to predict the behavior of complex systems. That's wrong. Briefly, Murphy's Law says, If anything can fail, it will. It was never intended to apply to forecasts of behavior, but rather as a ground rule as a design standard. Mr. Murphy's son recently clarified this in a public statement. He said:
I would suggest, however, that Murphy's Law actually refers to the CERTAINTY of failure. It is a call for determining the likely causes of failure in advance and acting to prevent a problem before it occurs.
snip
Murphy and his fellow engineers spent years testing new designs of devices related to aircraft pilot safety or crash survival when there was no room for failure (for example, they worked on supersonic jets and the Apollo landing craft). They were not content to rely on probabilities for their successes. Because they knew that things left to chance would definitely fail, they went to painstaking efforts to ensure success. EDWARD A. MURPHY III, Sausalito, Calif.
If Murphy's law did apply to real complex systems, we never would have had a successful rocket, or airplanes or computers to program in the first place. They would almost never work.
In practice it comes down to fuzzy language and imprecise logic.
If a discussion mixes improbable events with probably ones, and is vague about the consequences, i.e. the difference between anything fails and everything fails, then the chances of misleading everyone (including yourself) abound. In non-technical discussions, including this one, we can't resort to the only precise description (i.e. fault trees and their mathematics). That would be unsuitable for general audiences.
snip
Systems like power plants, airplanes, and automobiles are engineered systems. They are designed to provide a primary function. What is necessarily to accomplish the primary function is more a matter of physics than human preferences. Power plants make power, vehicles move things. Even if they have many secondary characteristics, their success is judged only on their ability to fulfill their primary function.
snip
Engineered systems tend to be very robust in fulfilling their primary function. Like the Apollo rocket, they can tolerate lots of failures and still succeed. Basically, everything must fail, or be rendered moot, for the engineered system to fail. Human-created artifacts, like a city are thought to have failed if any of the important functions fails. Basically, if anything fails, we rush to judge the system as failed.
The difference is profound. In Y2K discussions involving both kinds of systems, we must take particular care to keep this in mind. Software can be right or wrong.. Whether or not a system is considered failed because of wrong software is more a function of expectations than a function of logic.
********
http://www.albany.net/~dmills/fallback/postfad.htm The Post-Fad Era
Do you recall the Anteater cartoons? The ants were pictured as a little clump of dots who would scoot around independently but nevertheless as a clump. The image reminds me of the of the American paparazzi. The media hacks that jump from one hot story to another. Think of the Tonya Harding story, Lorena Bobbit, TWA 800, prom queen babies, not to mention O.J. Simpson. My point is this. Some day when there is no other compelling story, the paparazzi will pounce on the Y2K fiasco as the story of the moment. Scoot-scoot the clump of dots moves to the Y2K neighborhood. It becomes the lead story on every TV program, and the front page of every newspaper for 10 days. After that, the world will never be the same. Every TV set owner will be briefed up to the ears in the nuances of Y2K. I can't predict precisely when this event will happen, but I am certain that it will. Since the maximum news value and financial gain would come if the story breaks in 1999, I'm going to assume it will happen then.
The passage of this event will fundamentally change the nature of the Y2K problem and how society deals with it. Instead of today's sleepy neglect and disinterest, I expect society to flip to the other extreme. Hyperbole and hypersensitivity to everything Y2K related. It will be especially appealing to reporters and politicians. Let's refer to it as the post-fad era.
Will we have scores of citizens deprived of electric power because of zillion dollar invoices in the post fad era? I think not. Indeed I think it will become the biggest boon for deadbeats ever invented.
snip
Common sense tells us that the effects of publicity, fear and demagoguery, will aggravate some Y2K problems and mitigate others. Financial institutions like banks are likely to suffer the negative effects.
See Fallback, Chapter 7. On the other hand, I foresee that the embedded systems will generally benefit positively from the publicity.
Embedded controllers have some special properties. One weakness is that the software is embedded in ROM chips or other permanent storage that is difficult or impossible to change. Think of the chip in the watch on your wrist. If it has a Y2K bug, how will it be repaired? The answer is obvious; it won't.
On the other hand, these systems also have the fortunate property of being duplicated in many identical copies. That means that the manufacturer can examine the product for Y2K bugs. If found, he can alert all customers in advance. The cost of the Y2K investigation is inversely proportional to the number of copies. Casio watches and automobiles, for example, are sold by the millions.
snip
Advanced warning allows owners to take precautions to eliminate or mitigate the consequences of the bug. Consider a time-overcurrent relay from Acme Electric, (part of the set of protective devices for electric power equipment.) Acme can examine its code for Y2K problems in advance. Considering all the publicity it would be imprudent not to. In many cases the examination can be cursory. For example, it the time-overcurrent relay does not use dates and has no calender chip, then it can't be Y2K vulnerable. Even where a bug is discovered, advanced warning can reduce the consequences. In this example, it is likely that the bug would only be manifested on the dot of midnight 1/1/00. It could be avoided by hiring a worker to be there on that New Year's Eve and to unplug the relay one minute before midnight, plug it back in one minute later, and go home. Alternatively, a software update ROM might be sent in advance, or the unit replaced by another brand. One can imagine lots of precautions that would mitigate the damage. Further, if any of these precautions were satisfactory, then it would be insane to even consider modifying the software.
Online regulating systems are generally designed around the principle of nested feedback loops. Just the kind of feedback loops Fallback amply describes in Chapter 2 regarding the ripple effect. In nearly all important installations in all industries, these control systems have two important properties. First, each loop is able to be independently switched to manual or automatic modes. Second, the overall system is supervised by a human being, the operator. Let's consider how they would be used to mitigate Y2K ill effects.
In a large facility like a power plant or an oil refinery, the prudent thing to do on New Years Eve 1999 would be to call in extra help, and to put all feedback loops on temporarily on manual. After midnight, the operators could begin putting them back into automatic mode while monitoring their operation. Things which seemed to behave anonmalusly would be left in manual control. This is hardly unheard of or even unusual in many facilities. There are other circumstances when things are switched to manual as a precaution.
The effect of these precautions would not be to eliminate all the Y2K bugs, but it would detect and eliminate many of them. Now consider the ripple effect. The interdependency of many feedback loops steeply increases the exposure of complex systems to common mode problems. That's the ripple effect in a nutshell. But it cuts both ways. The curve is just as steep in the reverse direction. Each feedback loop placed on manual control, eliminates not only the local bug, but also ripple effects. If simple precautions allow us to simply avoid the consequences of, say, 80% of the hidden bugs, then the ripple effect says that the risks to the overall system will be greatly reduced. Perhaps a million fold less.
I'll say it once more just to be clear. The same ripple effect that amplifies the exposure of complex systems to common problems, like Y2K, also amplifies the beneficial effect of precautions taken to avoid the consequences. In the post-fad era, it is only reasonable to expect maximum efforts to take precautions. Because of that, I expect the glare of publicity to do more to mitigate the negative consequences of Y2K than the combined efforts of all us professionals to fix the software. Instead of making contingency plans to cope with Y2K effects, they'll plan to try to avoid them. I think, they'll largely succeed. It is a myth that computers control this world. Ordinary people still have the upper hand.
But, you ask, won't many people overreact, panic, and cause general chaos and disruption in reaction to the publicity? I'm sure they will. That too is attributable to the overall Y2K problem, albeit indirectly. Exploring the effects of public reaction and overreaction means there is yet another Chapter of Fallback remaining to be written. Sigh. Dick Mills, http://www.albany.net/~dmills/ West Charlton, NY 8/12/97
-- Cherri (sams@brigadoon.com), September 19, 1999
What's your point Cherri?
-- no talking please (breadlines@soupkitchen.gov), September 19, 1999.
You better change soup lines, they must be putting some of that funny stuff in your soup!! THE POINT IS that there are other opinions and other experts out there and we should get ALL SIDES TO THE STORY.Taz...shaking her head!
-- Taz (Tassie@aol.com), September 19, 1999.
Dick Mills is a great example of someone who cannot see the forest because of the trees. Hopefully he is right, but I sure am not going to chance it, especially given the interdependent nature of Y2K with regard to suppliers.
Thanks, Cherri. Out to buy more diesel fuel. (Before the prices go up even more!)
-- Jack (jsprat@eld.net), September 19, 1999.
Don't get testy Taz.The point of my asking Cherri what is her point, is to find out what her point is? Is there some current purpose in presenting this at this time? Was this article in some way missed previously? How does it relate to current news or articles? Is it being posted as a personal jab at Ed Yourdon or intended to be so? What is the point?
I believe that's clear enough.
To speak to your point concerning getting expert opinions, I wholeheartedly agree !!!
I found Dick Mills article interesting when I read it originally. I agree with your inference that opposing viewpoints and sources of information makes the debate more interesting and enlightening.
I couldn't agree with you more.
Point well taken, thank you.
-- no talking please (breadlines@soupkitchen.gov), September 19, 1999.
I notice that "can't see the forest for the trees" is becoming a code phrase for "actually knows what he's talking about."Take it from Big Dog -- experience is valuable only if you have it, in which case it's crucial. If you lack relevant experience, this lets you see the forest. Experience lets you see the trees, which only confuses you (unless, of course, you're experienced).
-- Flint (flintc@mindspring.com), September 19, 1999.
Anyone else lose track of the "Snips" and forget who was talking? The point BEING breadlines that whatever side you're on you're boring if you don't know when to shut up. Thank You.
-- Jimmy Bagga Doughnuts (jim1bets@worldnet.att.net), September 19, 1999.
Robust or Fragile? Mills thinks the electrical system is robust. Others, with just as good credentials, disagree. This last summer we saw, again, the inability to deliver all the demand at a full 120 volts. In my own area, the voltage was down to 107 for long periods. That is not a robust system. That is a system hanging on the edge, even without a lot of generators "down" due to fuel or failures.
-- Gordon (gpconnolly@aol.com), September 19, 1999.
There are SO many variables. I would LOVE it if the utilities stay up!!! But even if they do (I hope, I hope) what about other systemic failures? Got to prepare regardless....
-- Mara Wayne (MaraWayne@aol.com), September 19, 1999.
No, Flint, the phrase applies to someone so caught up in specific details of what they happen to (for the sake of argument) be so well versed in, that they ignore everything else that might apply. You know, like you rambling on and on about how to flip bits so that the OS of a PC will be compliant, while never realizing that its the applications and data thare are the real Y2K killers.
-- King of Spain (madrid@aol.com), September 19, 1999.
KOS:You mean when I keep saying the problems lie in the applications and not the firmware, I'm failing to recognize that the problems are in the applications? Far out...
-- Flint (flintc@mindspring.com), September 19, 1999.
Sorry, Flint, I had no idea that you ever said that. Of course, in the middle of one of your NEVER ENDING stories, you could confess to being Jack the Ripper, and I'm sure that I would miss it. Nobody has the time to wade through all of your crap.
("Far out..."????? Gawd, you really date yourself, dude.)
-- King of Spain (madrid@aol.com), September 19, 1999.
Someone correct if I'm wrong, but I think this topic has been brought up before. Weren't the errors pointed out errors in the rough draft-- the rough draft on the Internet that Ed Yourdon was soliciting comments on and revising--and which were corrected before Time Bomb 2000 was ever published in book form?...As a matter of fact, I just pulled out my copy of the first edition of Time Bomb 2000 and noticed that Ed incorporated Dick Mills' objections into the first edition. Ed Yourdon quoted the objections of Dick Mills in Time Bomb 2000's first edition!
-- Linkmeister (link@librarian.edu), September 19, 1999.