Nothing Happens Until the Fat Lady Sings

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

In the sports days of yesteryear, a player or fan who would talk too much about what might happen in the next game was reminded that "Nothing happens until the fat lady sings"; meaning that you should prepare "the best you can" but you won't know what your facing until the game starts.

As a systems test engineer for weather satellites I learned, years ago, that one doesn't even have a clue as to the nature of one's problems until one is well into the testing phase. Not a clue.

Before each of our tests, the individual modules tested out fine. But the system never tested out fine right away. Never. Until the testing, until the fat lady sang, I swear we were clueless as to what the real problems were.

You can test banking systems, you can test billing systems, but testing for power plants is different. Presently, the problem is approached by engineers saying that various components are being checked out and, from the positive status of these components, they are predicting success.

Add to this that a realistic systems test would include tele vendors and their part of the system as well. Not only is "full up" realistic testing being avoided but unrealistic testing is not yet even occurring!

Our problems go way beyond software and component remediation and testing.

For example, there is a trap, into which, we have fallen. We presently divide our progress board into the stages of: Inventory, Assessment, Remediation, and Testing. But the first two don't count for much. And Remediation results can be misleading becauses one won't know if he has remediated everything until the testing starts.

We are using a four stage paradigm that does not tell us if we are making progress. This paradigm could be blown to bits when the testing starts. This four stage paradigm even tricks into believing, like NERC does, that we are right on schedule. After all, many plants have completed their inventory and are well on their way to getting their assessment done. "Heck, looks like were half way there."

Ya, right on NERCers, we doing just fine.

Our four stage paradigm keeps us in delusion. Here is a cut at a more realistic paradigm that keeps us honest:

I. Inventory, assessment, and remediation. II. Sub-system testing. Fix what you find. Retest. III. Regression testing. (Once A, B, and C work then get A and B working together and then A and B and C working.) Fix what you find. Retest. IV. "Full up" testing without tele. Tele testing. Then both together. Fix what you find and then retest.

Now, using this new paradigm, lets seet where most of the power plants are. Yup, incomplete on phase I. Hmm. What do the boys at NERC do now?

The problem lies not in too little time The problem lies in our paradigm.

-- Anonymous, November 03, 1998

Answers

Steve, that's a superb assessment and clearly shows why the reports the public are given will be illusionary unless the person reading them has a technical background. I would say, though, that too little time is a problem as well as the paradigm. Some of the programmers who were yelling and making personal preparations a year and half ago had already decided, using your paradigm, that most agencies and businesses were not going to get to III or IV, and there would therefore be no compliance guarantees which could be trusted. Here's a concrete example to illustrate why testing is mandatory:

I have a friend overseas who is managing a consulting team which is installing and integrating an updated "Y2K compliant" system. Here are excerpts from a letter. "...this is a (new) system that was conceived, designed, and built to be compliant from the get-go by an organization with a high degree of awareness....products that are in use by a healthy proportion of Fortune 500 companies... I just learned that the system in question has * X* (number deleted by me for legal and privacy reasons - it's under 100 and more than 10.) identified Y2K defects....On the one hand, this number is miniscule for the scope of the system....On the other hand, that there are any at all can only be attributed to habitual techniques almost subconsciously kicking in." How many of you reading this wrote '97 instead of '98 on a check the beginning of last January before you caught yourself in the error? Habits die hard.

You can't be sure of anything unless you've tested, and re-tested, and tested some more. Period, end of story. Considering the reports we do have, that leaves us in the Intermission of the Opera, wondering how badly the Fat Lady is going to sing at the end. Intermission is a good time to stock up on supplies, folks!

-- Anonymous, November 03, 1998


Superb post and right on with my own experience in tech matters. Will repost your pardigm so others can enjoy and learn.

-- Anonymous, November 04, 1998

Moderation questions? read the FAQ