Y2k Errors and Metrics: Is the Worst Already Over?

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

One of the main Polly arguments revolves around the fact that errors happen all the time, and are dealt with. Software systems definitely work at less than 100% efficiency today, and in all liklihood always will.

Typically, this is countered with Y2k being a singular event; that all systems will simultaneously deal with Y2k problems, and this is unprecedented. The "systemic" nature of Y2k and systems will then cause a collapse.

But, is it truly unprecedented? Maybe not. Maybe, we have already dealt with a number of situations, where the errors may have been even cumulatively greater. Yourdon, Hyatt, North, etc., all predicted dire consequences for pre-Y2k dates; Jan 1, April 1, July 1. However, they based these on such things as fiscal year turnovers, etc. Maybe, they were on the right track, but didn't take it far enough.

Let's start with the universe of un-remediated, non-compliant systems at the end of 1997. These are, afterall, the basis for all the doom predictions. A percentage of these are simply not fixed, and remain at risk. To be conservative, let's assume 1/3. So 66% of the applications will be replaced or fixed, and 33% will not.

Major implementations do not typically happen at random; at the very least, they tend to concentrate around month-end/month-starts. Note, this is conservative; most financial applications, which are the most at risk, tend to be implemented at quarterly intervals. But let's use each new month, giving 24 unique "events", where major implementations are being conducted, simultaneously.

The final assumption is the number of systems replaced, versus remediated. I have no reliable information here, so let's assume 50% are replaced. My experience indicates a larger percent, but it is biased, since I do work with SAP implementations.

So, let's look at a baseline error rate for rollover. In the past, I've used metrics by Howard Rubin regarding date logic. Here, let's use even more conservative numbers, from Capers Jones, available here:

http://www.year2000.com/archive/proby2k.html

OK, from Capers Jones, approximately 15% of function points contain date reference. Given the above, 33% of these remain untouched. So 5% of the function points contain untouched date references.

In addition to these, Capers Jones estimates that even in remediated code, a 2.25% will contain errors not fixed, or new errors introduced through the fixes. Leaving a total of 7.25% of function points containing invalid date references or new errors.

But, not all date errors occur on rollover. Gartner Group has in the past estimated 8% of the date errors happening at rollover. Again, let's be conservative, and use 20%. So 20% of the errors still in existence will occur at rollover, the rest spread overtime. This leaves a baseline of 1.45% of function points having errors at rollover.

So, we have our baseline, of 1.45% at rollover.

What of the 24 events from 1998 thru 1999? First, let's distribute the other 75% of errors in remediated applications. This leaves approximately 5.5% of function points with date errors. Again, very conservatively, let's assume half occur prior to rollover, and half after. So 2.75% of the function points have errors that occur prior to the rollover. Spread out over our 24 events, leaves .15% per event.

That deals with unremediated applications, and remediated applications. To this, we add new development, or replacements. The latent error rate here is much worse, as would be expected. Errors can occur virtually anywhere, and usually do. Capers Jones estimates that 75% of function points within new development contains errors. In our universe, 66% of these were fixed or replaced, leaving 50% of the function points having errors, in fixed or replaced systems. And, using our assumption that only 50% were replaced, leaves 25% of the function points containing errors due to system replacements. Spread out over the 24 events leaves 1.04% of function points containing errors due to system replacement, at each event. Adding in the errors due to implementation of remediated systems, gives 1.2% of function points in our universe containing errors at each of the previous 24 month-ends, beginning in 1998, and running through 1999.

So, 1.45% versus 1.2%, 24 times. And these are very conservative numbers, folks. I assumed a uniform distribution of implementations over 1998 and 1999. But realistically, my guess is they have been weighted towards this year, and in particular the June 30th time frame. These are the errors which have been at times noted, and span the gamut of possibilities. But again, 1.45% versus 1.2%, 24 times. Doesn't seem quite so unprecedented anymore.

-- Hoffmeister (hoff_meister@my-deja.com), July 31, 1999

Answers

So your point is that Y2K as an event is over, and it only gets better from here, based on numbers extrapolated BEFORE Jan. 1?

You go ahead and believe that. Enjoy.

-- INVAR (gundark@sw.net), July 31, 1999.


Thanks Hoff, any chance that since the main thrust of y2k is over you could pack it up and head onto your NEXT assignment??

Your Pal, Ray

-- Ray (ray@totacc.com), July 31, 1999.


No, INVAR, my point is that as far as software errors are concerned, we may have already experienced, and survived, a number of cumulative events that outweigh the actual rollover.

-- Hoffmeister (hoff_meister@my-deja.com), July 31, 1999.

Hoofmeister,

Interesting analysis, but I would expect the number of errors not to be uniformly distributed over your 24 events, but to be clustered somewhat more thickly around those events in the range of rollover plus or minus 3 months, with fewer errors the further away from rollover in either direction. NOt quite a bell curve, but certainly not an even distribution.

Either way, you have presented an interesting hypothosis that may very well have some validity to it.

-- Paul Neuhardt (neuhardt@ultranet.com), July 31, 1999.


But we haven't even hit the rollover yet Hoff. The only way anyone is going to know if your hypothesis is correct is compare numbers garnered AFTER rollover and compare that to pre-rollover numbers.

Should we all wait around and hold our breaths to see if that is so?

I'm not. Even if you turn out to be correct.

-- INVAR (gundark@sw.net), July 31, 1999.



Hoffmeister:

Well, it's nice to see that you've given the issue some serious thought.

However, there are substantial errors in your flow of logic; rather than spend the rest of my weekend pointing these out, a pair of examples should start you on a reconsideration:

"Let's start with the universe of un-remediated, non-compliant systems at the end of 1997. These are, afterall, the basis for all the doom predictions. A percentage of these are simply not fixed, and remain at risk. To be conservative, let's assume 1/3. So 66% of the applications will be replaced or fixed, and 33% will not."

This is the premise upon which your other figures are based. The figure itself has no basis -- you have "conservatively assumed". You cannot base a statistical argument upon assumptions. This figure must be based on some kind of evidence for the remainder of your analysis to have ANY weight at all. In view of the number of businesses and agencies whose remediation is STILL not complete, your assumption that by 1997 66% were done is highly suspect in any case.

Further, even though your original number is supposedly based on "the universe of unremediated" systems, by your statement that "...financial applications....are the most at risk...", you indicate your actual primary focus: accounting. Additionally your reference to the JAE dates establishes your focus: accounting. If you are speaking strictly of "applications" of software, you must realize that software operates more than accounting programs throughout the business world -- and these potential failures will do more than mess up balance or spreadsheets. I am sure that you can think of numerous software applications that are not accounting/fiscal/yearend related, and will not require a basic list. These must also be factored into your figures before you can draw any conclusions on the rate of failures -- a company may have a "perfect" fiscal-year-end rollover, and still be unable to sell its product because it can't get a particular machined component from a supplier whose device-control software has failed.

To sum these two points: (1) your figures are imaginary (note the number of times "assumed" is used, and (2) you have limited your argument to specific software applications (hardware isn't even addressed).

Consequently, based on ONLY these two errors, you cannot legitimately come to the broad conclusions you express. Further erroneous assumptions weaken your argument even more -- but these two alone are sufficient to establish your error.

Anita Evangelista

-- Anita Evangelista (ale@townsqr.com), July 31, 1999.


Granted, there are some guesstimates. But they are based partly on experience, and partly on other actual figures.

For example, you pick out the 66% being remediated or replaced. Not completely without basis. Take the Cap Gemini survey, recently quoted by Hyatt. Some of the data can be found here:

http://www.usa .capgemini.com/news/pr99.asp?id=86

In it, they find that 78% expect to have more than 76% of their code completely tested and compliant. (Note: this has been falsely reported as 22% not having their "mission-critical" systems done, a term that doesn't appear in the Cap Gemini information). So I used 66% unremediated or replaced, which, while an estimate, does fit within the range of the Cap Gemini survey, and is sowmewhat conservative.

As for financial apps, I mentioned these because they certainly tend to cluster implementations around quarter ends, and were the focus of the Hyatt, Yourdon, North predictions. I did not, however, use these for the estimates, but instead went to the month-end/month-start events, which again, from my experience, represent the bulk of implementations, financial or not.

-- Hoffmeister (hoff_meister@my-deja.com), July 31, 1999.


My points, exactly, Hoffmeister.

You said: "Granted, there are some guesstimates. But they are based partly on experience, and partly on other actual figures."

Have you searched for other figures that might contradict your "guesstimates"? If not, you are being intellectually dishonest. And your results are still in error if they are based on guesses.

Next you quote Cap Gemini: "In it, they find that 78% expect to have more than 76% of their code completely tested and compliant."

Finally, note key word: EXPECT. I wonder how many people who plunk down $2 EXPECT their lucky number will win the lottery? (Hint: all of them.)

Your results would be more reliable if you, say, compared "expected compiance" by June 1 with ESTABLISHED compliance (3rd party testing), and then used that figure....

Without serious numbers, your results are nothing better than fantasy.

Anita Evangelista

-- Anita Evangelista (ale@townsqr.com), July 31, 1999.


Hoffmsiter, And here's a big one you didn't add in--that no one sees your point and millions of people still panic in October, November, December. The food supply disappears and the market crashes. So even if the software is fairly okay (which I doubt to the max) then the situation is still devastating and lousy.

-- Mara Wayne (MaraWAyne@aol.com), July 31, 1999.

Yes, Anita, these are estimates. They do have basis, but are estimates, nonetheless.

All of the prophecies of "doom" are hypothesis, based on estimates. I purposefully used the same sources for my estimates. I could definitely find much more optimistic numbers, but did not. Without exhaustive research, I try to find the most conservative estimates and metrics. Basing the numbers on estimates such as these should give a wider margin for error.

To Mara, people have predicted "panic" for countless months. Continuing to base actions on predictions of supposed "panic" by the "sheeples" gets further and further from reality.

-- Hoffmeister (hoff_meister@my-deja.news), July 31, 1999.



With five months to go, how many Fortune 500 companies have now finished work on their mission-critical systems?

-- Linkmeister (link@librarian.edu), July 31, 1999.

Hoffmeister said: "Yes, Anita, these are estimates. They do have basis, but are estimates, nonetheless. All of the prophecies of "doom" are hypothesis, based on estimates. I purposefully used the same sources for my estimates."

But you also said: "But they are based partly on experience, and partly on other actual figures."

Which of these statements are correct? "Same sources" as others use, or "based on (your) experience"?

These statements are mutually exclusive.

It's very nice that you are able to use "experience" to justify your arguments -- but if a "doom" hypothesis uses "their experience", you will readily discount their conclusions. Somewhere, there is a serious disconnect between what you want others to do, and what you yourself do.

If you truly wish to provide any accurate statistical basis for your original hypothesis, you MUST use accurate figures -- not "experience". This is the foundation of statistics, buddy. You have ignored this basic fact. Therefore your conclusions are unreliable.

No amount of explanations, disclaimers, or other distortions alters the fact that your initial premises are in error.

Anita Evangelista

-- Anita Evangelista (ale@townsqr.com), July 31, 1999.


Sorry, Anita, but my experience and other sources are not mutually exclusive.

But, to help you out:

The 66% of application being remediated or replaced is a guesstimate. It is based both on my experience, and backed up by the Cap Gemini survey, quoted by Michael Hyatt.

The fact that implementations tend to cluster around month-end/month-start is based on my experience. If anyone disputes this, fine. But it is what I've encountered.

Finally, the 50% of applications being replaced is my estimate. Again, my experience is this number is much higher. But, realizing my experience is skewed, being in SAP implementations, I chose a more conservative number. If you have better information, please share.

The rest is based almost solely on Capers Jones' metrics, a source used often by Ed Yourdon.

The other source is Gartner Group, in their estimate of 8%. Again, this I made much more conservative by increasing to 20%.

-- Hoffmeister (hoff_meister@my-deja.com), July 31, 1999.


also, please note that all statistical arguments are based in many ways on assumptions. For a few examples, we assume that what we are measuring is what we intend to measure, that correlations we observe will continue to hold within certain constraints in the future, that any metrics we factor in are valid and accurate within certain limits, that we have properly determined the shape of curves (you can find a statistical 'best fit' for a curve of *any* shape), etc.

Hoffmeister's basic argument is that the curve of incidents of bug- encounters is 'squashed' fairly flat (the kurtosis). This implies (1) that we've already encountered a sizeable percentage of them; and (2) that the remainder will be encountered over a fairly extended period of time. Since those we've already encountered have been handled with few difficulties, it remains entirely possible that the *rate* of bug incidence will fail to rise above our ability to deal with them within any given period of time.

Essentially, to argue otherwise you must establish that Hoffmeister's assumptions are not reasonable. Since his assumptions are based on the most solid measurements available from the best available sources of quantitative data, you need to either (1) supply data from other sources *at least* as well documented showing something very different, or (2) reject his analysis on the grounds that it is incongruent with your belief system.

-- Flint (flintc@mindspring.com), July 31, 1999.


Hoffmeister, Panic comes closer to reality, sorry (very), but no, we can't roll the clock forward or backward. Time in our limited linear sense doesn't work that way.

-- Mara Wayne (MaraWAyne@aol.com), July 31, 1999.


Hoffmeister, I'm not an IT person, but isn't it possible that we have been largely sheilded from what would have resulted from the 19 events of 1998 and 1999 that we've already encountered, by fixes that tell computers not to look ahead just yet? Sometimes that's not possible, but I suspect it is for a large percentage of applications. And isn't it possible that we've been sheilded from errors caused by code remediation, simply because a large percentage of the remediated code is not yet on line and running? It has appeared to me that most of the "Y2K errors" being reported this year are not caused by 2000 dates, but occur quickly after remediated code is put back online. These failures are "side effects" of remediation. If they were caused by 2000 dates, the unremediated code they are replacing would already have failed. Finally, isn't it possible that, using the logic you used above, even a complete worldwide failure to remediate ANY code would not cause an alarming percentage of errors relative to what you were using? In other words, if your argument above is valid, a worldwide decision to fix on failure may well have turned out to be more cost effective.

-- Bill Byars (billbyars@softwaresmith.com), July 31, 1999.

WOW, did I sleep late, and miss the last 5 months? Ya know, they call it Y2K for a good reason, not the "various dates in 1999" problem...

Where's my RUBBER STAMP...

Duh, or is it Doh? <:)=

-- Sysman (y2kboard@yahoo.com), August 01, 1999.


Links here on whether and how significant the Jo Anne Effect is:

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=00122f

"Significance of States Fiscal Start"

-- Linkmeister (link@librarian.edu), August 01, 1999.


Kev, bud, you miss the point.

Very few, if any, of the errors I'm talking about are "JAE" or look-ahead.

Bill, for the most part the errors being discussed are just what you describe, secondary errors due to implementations.

No, I don't believe we'd have been better off not fixing anything. Some of the potential Y2k problems in software are not the type to be easily fixed. But, in essence I do think, in total, many more errors have been and will be generated due to implementing and replacing such a large number of systems globally, than will be generated due directly to Y2k problems.

Sysman, you might want to read first. Or not, the choice is yours.

-- Hoffmeister (hoff_meister@my-deja.com), August 01, 1999.


So to answer your question Hoff - an emphatic Nooooooooooooooo!!!!!!!

-- Andy (2000EOD@prodigy.net), August 01, 1999.

Gee, Andy, nice to see you've given this your usual well-thought opinion.

-- Hoffmeister (hoff_meister@my-deja.com), August 02, 1999.

Moderation questions? read the FAQ