Hoffmiester I have read your rebuttal of Ed Yourdon's letter to Alan Greenspan. As one of those grunt programmers I wish to take issue with your view of Y2K and also ERP packages in general.

As regards the Y2K issue:

Reviewing and fixing software for Y2K bugs is not a simple straight forward process as you would lead us to believe. In our own system which I wrote over a ten year period dates are used in many different ways and formats. They are often burried in code that does not reach out and scream I am dealing with dates. Failure to fully understand how certain lines of code are affecting dates can lead to bad data passed on to other sections of code, other programs and database fields. This is what I experienced in remediating software that I wrote. Lord help those who are doing Y2K remediation on systems they did not write themselves and are most often much larger systems than is mine.

Sure I will grant that inperfect code results in failures each and every day. However there is a point of critical mass that can happen if enough failures happen at the same time. At this point a downward spiral can occur which overwhelms the human resources available to restore orderly functioning of our economic and life support infastructure. The valid point of discussion as I see it is are we going to reach critical mass, I think we will, and if so how far down will we spiral, I have no idea. We will soon find out.

Perhaps what has concerned me the most, led me to believe that Y2K will be much more than a bump in the road is the lack of concern and understanding by those in management positions as well as IT itself of the ramifications to society if the Y2K issue is not dealt with headon. Sometime back I intervied with the COO of a mid sized company for a programmer / analyst position. I asked him if they had a handle on in house Y2K issues. He nodded in affirmation. This should have been my clue. I accepted the position, my mistake, and soon found out through my own testing that the existing software package that this company had installed with several hundred customers would roll over and die on 01/01/00. I pointed this out in a staff meeting and was met with yawns. I then pointed out that this situation could lead to lawsuites. This is what got their attention. The COO and others in management were were Y2K clueless. The golf course mentality was well entrenched. The golf course mentality is well entrenched almost everywhere. Y2K is not a game of golf. A confident mentailty by management and IT personal that Y2K is not a big problem does not make it so. Only grunt work can turn Y2K into a non problem or a small problem. Time is short and grunts are scarce.

On ERP packages:

A good number of organizations are now in the process of implementing ERP systems and have to be live by 01/01/00. It is do or die for them. They have determined that their existing systems can not be Y2K remediated by the end of this year. Under such a time constraint some organizations will not suceed. It boils down to 'if the right hand don't get you the left one will'. A successful implementation of ERP requires dedication and participation by all users in the organization and also from the ERP vendor. Implementing ERP under Y2K duress is a prescription for disaster.

ERP is not for everyone. Although it is not quite a one size fits all it is a close cousin. My limited experience with ERP, not the SAP package, is that much more maintenance is required by the users to maintain pricing, commissions and other basic tables which are the nuts and bolts of an information system. Autodup, mass copy and edit features that a grunt programmer learns to include in a system to make it more user friendly and efficient are sorely missing.

I assume ERP systems are all Y2K complient. If not then this alone is a major problem.

Final thoughts:

There many readers of this forum who are not programmers, engineers, etc. and therefore must rely almost completely on the word of those who are as well as what manages to make the news in order to form an opinion on the consequences Y2K will have for them and their loved ones. If they make provisions for Y2K disruptions or worse and it turns out to be only the much promoted bump in the road then their food bill will be that much less next year and those friends and family who laugh at them next year will soon forget.

If however Y2K is a severe event then those who did not prepare but were warned either by this forum or other avenues may face the prospect of watching their children starve. This is not a legacy I wish for myself and neither should you.

As an aside some people have in their Y2K preparations discovered an entirely new lifestyle of more independence and accquired basic skills they will never forget. For them there is no going back Y2K or not.

-- Ed (ed@lizzardranch.com), September 23, 1999

Answers

Ed,

You got my vote!

This is the one area where I have a big problem with sir Hoff and his "we're dealing with it now" theory. He seems to think that we are experiencing Y2K in 1999. Yes, systems are being fixed or replaced now, and this is introducing other problems. What he doesn't realize is that these "other" problems only add fuel to the fire. Are we dealing with the problems now? Yes, but I said it earlier today, and I'll say it again.

Programming is all about choice, A or B, C or D. Except for the tiny percentage of programs that do look ahead processing, the "00" choice isn't here yet. Hoff thinks that we will be able to deal with the mountain of problems that are about to land on us. I'm not so sure. In fact, I'm almost sure that we can't.

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), September 23, 1999.

Ed,

Thanks for responding to Hoffmeister's last missive. I've been tied up this week with "ordinary" (non-Y2K) work to pay the rent, and haven't had the opportunity to keep up the dialogue.

As several people have suggested, though, it's likely to be an endless battle -- and I was prepared to write a short note to Hoffmeister suggesting that we simply agree to disagree.

Thanks again...

Ed

-- Ed Yourdon (HumptyDumptyY2K@yourdon.com), September 23, 1999.

Mr. Yourdon,

Yes, we could "agree to disagree" but then what are we going to do here for the rest of the year? Post the "latest" Y2K news, and argue about it?

I'm here because I've found another programmer (you) that shares my view of the Y2K problem. I respect your opinion because you've been involved in the computer business longer that I have (35 years vs. 31).

I salute you for bringing this problem to J. Q. Public via your book. I don't care how much money you have made from your book, or other projects; you don't have $.01 of my money. You had the balls to bring this problem to J. Q. and let him deceide on his own.

Ed, it ain't over yet. I appreciate your Humpty view, and it does have it's place. But, if we can make only 1 or 2 or 3 more GIs, before the drop-dead date, isn't it worth it? Won't these 1 or 2 or 3 be part of the solution, and not part of the problem?

Give Sir Hoff HELL!

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), September 23, 1999.

Actually, Hoffy provides a valuable service to everyone, since I think that it is fair to say that he is probably the BEST spokesperson for the polly point-of-view that we currently have. (Paul Davis has been thoroughly discredited, Flint has gone off to la la land, Ken Decker is just plain becoming a nice old softie.) And Hoffy's arguments, though loaded with seemingly exhaustive statistics up the ying yang, completely flop once common sense is applied.

Especially when one considers that the bottom line issue, for anyone, is really: "Should I play it safe and prepare, or should I just throw caution to the wind and not worry?"

-- King of Spain (madrid@aol.com), September 23, 1999.

Hoffmeister's basic argument that we are now experiencing error rates comparable to what will occur at rollover is intriguing. I've enjoyed following his debate with Mr. Yourdon, which also directed me to Hoffmeister's past debate with Steve Heller, which I had somehow missed. I've questioned Hoffmeister on a couple of points in the past, but I've always found him eloquent and willing to do a lot of research to support his arguments. Those are also qualities of certain other posters on this and other fora, with widely differing views on Y2K, apparently.

As I've admitted before, this isn't my field: my graduate training in English at the U. of Illinois did not include any courses in computers or software metrics that I can recall. Eight months ago I read a couple of books by Capers Jones on software metrics, including "The Year 2000 Software Problem"; for the past 18 months I've followed (as have quite a few other people on this forum) the reports by GartnerGroup, Cap Gemini, Yardeni, the OMB, GAO, Senate, President's Conversion Council, UN, World Bank, BIS, CIA, etc., along with reading an assortment of on-line and traditional newspaper and journal articles, plus innumerable internet postings (of widely varying quality and reliability, it would appear). I do have some background in math and statistics (calculus in high school; two years of grad business school later on), and a halfway decent memory for numbers. What I don't know is anything about programming and systems analysis--or about IT systems per se, how much companies rely upon these systems, how easy it is to do manual workarounds, or a host of other important, concrete issues that obviously only a professional working in the field would be in a position to begin to answer.

All that confessed, I'm still going to make a few observations on Hoffmeister's software metrics arguments. Feel free to hoot. My father always said that I should make something of myself, and no doubt I am about to make a fool of myself.

Hoffmeister, your mathematical calculations are impeccable. And you go out of your way to grant some "worst case" scenarios in an effort to build a "safety margin" for your analysis. But there might be several weaknesses in your analysis, too.

First off, the crux of your argument is the supposedly very high rate of errors in implementation of new software systems (SAP and such) designed to replace Y2K-plagued software systems. You assume that of all active software programs that will be handled for Y2K, 50% of them will be replaced by SAP, etc., instead of being remediated. That seems like a considerable assumption to me, especially since 20,000 SAP installations to date seems like a relatively small number in the context of total sizeable companies worldwide that theoretically could use such systems. You note in passing that a review of SEC filings will show SAP mentioned with some frequency by Fortune 500 companies. Do you have any hard numbers or documentation--from Jones, Cap Gemini, GartnerGroup, etc.--to demonstrate just how widely SAP and similar programs are being implemented in major U.S. and overseas companies? I recall reading last year (sorry, can't remember the source) that most companies were not opting for the SAP route but were in fact going ahead with actual remediation of most systems (or at least the "critical" systems). Now, that plan may have changed since then, as more and more companies realized they didn't have time for remediation; but I think we still need some documentation here. There are a couple of related questions, too. To what extent are these various companies that are going the SAP route replacing their systems (at least the mission-critical ones) with SAP? I don't know anything about SAP systems, obviously, and would appreciate any info you care to volunteer here--but I note that you and Mr. Heller disagreed about the extent to which SAP is typically used within a given company. Again, is there any hard data here? Also, with regard to the frequency of such installations (key to your metrics argument): for your calculations, you simply divide the number of SAP and similar installations (or more accurately, the errors arising from such installations) by 24, as though this data were spread out evenly (at month's-end intervals, as you note) over that 1998-99 period. Yet elsewhere you note that SAP installations (Y2K-related?) were beginning as early as 1994. (Major banks, for instance, started their Y2K work in 1995 or thereabouts.) Shouldn't your calculations be based on a 5-year period, with, granted, much more weight given to the last two years? And perhaps in any such calculations, a great deal of weight should be given to the last few months of this year, as possibly a large number of laggards rush to finish installing these systems. As you know, GartnerGroup does not project significant number of Y2K-related problems to begin until October. Is one basis for this Gartner projection the possibility that implementation errors arising from new (replacement) software will increase rather significantly in the months immediately ahead?

Next, regarding Capers Jones's estimate of the error rate in new software: this is probably the heart of your argument, but I think you might have pulled the wrong number from Mr. Jones. You state in your debate with Mr. Heller: "Jones estimates that 5 errors per function point are introduced through new development." Now, you are very gracious about discounting many of these new development errors vis-a-vis Y2K errors; in fact, the Hoffmeister discount rate is 85%, which bolsters your argument considerably. Still, it might be noted that Jones's estimate of 5 errors per function point (FP) was *before* vendor delivery to end user. In your debate with Heller, you cite Mr. Jones's article, "Probabilities of Year 2000 Damages," the URL for which is http://www.year2000.com/archive/proby2k.html In Table 1, "United States Averages for Software Defect Efficiency," Mr. Jones presents the data I just mentioned. You looked in the "Defect Removal" column (where the "5 errors" number is given; it seems to me that you should have looked in the "Efficiency Defects" column instead. (You did look in the correct column in the next table, when considering Y2K errors and removal efficiencies.) After all, Mr. Jones's point apparently is that the vendor/developer will remove 85% of these 5 errors per FP, leaving .75 error per FP, BEFORE delivery to the customer for implementation. In other words, the "Delivered Potential" is 85% removal of errors. By using 5 instead of .75, you have exaggerated the implementation error rate by a factor of almost 7X, which, obviously, would alter your subsequent mathematical calculations noticeably. It means that, all other factors being held constant, the rollover Y2K error rate would not be roughly equal to the current error rate (mostly from new implementations) but, in fact, would be almost seven times greater then the current rate. And since the whole is often greater than the sum of its parts, both for better AND for worse, one wonders if this 7X error rate next January might not somehow be a "critical mass" that would trigger events and problems going beyond simple computer system failures. As you may recall, I am not a "doomer"; my Y2K position is rather middle of the road, with my primary concern being longterm economic issues, not basic infrastructure ones. My present WDCY2K rating for the U.S. is very close to that of Bruce Webster, in fact, who, as chair of the Washington D.C. Y2K Users' Group has access to all kinds of "inside" info that I lack. In short, I do not expect anything resembling wholesale collapse. But, at least for me, it's also really hard to estimate just how much downside potential *might* be out there.

Back to the relative error rates. Am I missing something here? After all, you are a software consultant and know about such things, whereas I do not. Is a new software system normally just dumped (hot from the press, so to speak) on a customer's doorstep with all the raw bugs intact, and then worked out on-site, to the chagrin of all concerned? Or doesn't the vendor/developer (as Mr. Jones's "Delivered Potential" criterion suggests) first try to "edit/proofread" the software in-house, before delivering it to the customer for the arduous process of implementation? (Forgive me for expressing this in ways familiar to me as a writer.)

Again, I don't know, this not being my field--thank God. One reply on your side that I can imagine is that SAP is somehow, in effect, mainly *developed* on-site, written on the spot, so to speak, to meet each customer's specific needs. But I always understood that SAP was basically a "one size fits all" type program--with some modification for each customer, of course. After all, I realize that SAP software packages aren't something you pick up at the software store, carry off to your favorite company, load in a few minutes, and go on your merry way. I'm under the impression that the SAP implementation process is long, difficult, and messy--hence the need for highly intelligent and worthy gentlemen such as yourself. It would seem to me, however, that there must be some actual, developed SAP software programs to install in the first place, however much adjustment is needed later: programs that have been developed and tested in-house by the vendor, and that have met some reasonable baseline for efficiency (Jones's "delivered potential" criterion yet again), before being wrestled into place in some corporation's overall IT structure. But I might easily be just ignorant of how such things work; for a nontechnical outsider, it's hard to visualize/conceptualize such matters.

If I'm not totally wrong here, however, then, indeed, you *might* have overestimated the implementation error rate by far more than a factor of seven. You see, Mr. Jones's stats were, as the Table 1 title suggested, "averages"--and software development efficiency varies greatly, by a factor of 10,000X or more, from company to company, as Mr. Yourdon has pointed out. Now, I don't know Mr. Yourdon's qualifications as an "expert" on software metrics, except to note that the Yourdon Computer Press Series (which later became part of Prentice-Hall, just as one of his companies became part of IBM) has published quite a few works on software metrics, including, I believe, some of Mr. Jones's books. It might not be unreasonable to assume that as chief editor of that press (along with being the editor of three major software journals and author of 24 books on computers) Mr. Yourdon might have read a few of those software metrics books. Also, he may have had some practical experience, since, as CEO of the Cutter Consortium (whose parent company, Cutter International, has some 20,000 client companies worldwide) he has worked with enterprisewide mainframe IT systems on every continent--except Antarctica, that is, unless the penguins have gotten computerized these days. And long ago, in conjunction with some fellow named Whitehead, he developed a little methodology called object-oriented design and analysis. I have no idea what that is, but you no doubt have heard of it; it's supposedly the most important software engineering advance of the past 30 years (or so says the "Encyclopedia Britannica"--but what does it know?). Regardless, I don't know Mr. Yourdon personally, just his resume, and I'm obviously not in a position to judge whether he really knows anything useful about enterprisewide mainframe IT systems and software metrics. And he may be all wrong about Y2K, as he himself has acknowledged. Still, let me quote a passage from p. 117 of the first edition of "Time Bomb 2000," for idle amusement:

"One of the most depressing statistics about the computer software industry is that its professional practitioners deliver allegedly well-tested software [the "Delivered Potential," in Jones's jargon] to their customers with an average of approximately one defect (often referred to as a 'bug') for every hundred program instructions. With good discipline, sophisticated tools, and sufficient resources, the average IS/IT organization within the typical American company can reduce the defect levels by approximately two orders of magnitude--i.e., to the level of one bug for every 10,000 program instructions. The 'best of the best' organizations--e.g., organizations like Lucent (formerly Bell Laboratories), Motorola, and the NASA Space Shuttle software group--can improve upon this by another two orders of magnitude, thus achieving a miniscule one bug for every million program instructions." [MCI WorldCom might disagree.]

You sense my question: if these other IT outfits, internal or external, can reduce the error rate by a factor of 100X or more, then shouldn't we expect that SAP, a big company/vendor, I understand, with lots of experience and expertise, would be able to do likewise, and get its new development error rate (again, the "delivered potential") down by a factor of 100, as compared to the shoddy "averages" Jones cites (which presumably include all sorts of software companies, great and small)? And shouldn't the same be expected of SAP's major competitors in this rather lucrative market? Or is there something I don't understand about the way these companies operate? Is SAP really so shoddy as to deliver software to customers with one bug per 100 LOC?

But actually, if Jones's "average" stat were used, SAP's record would be worse yet, because Jones's metric is expressed in terms of FP, not LOC. Let's see, Jones's "average" stat was (again, based on "delivered potential") .75 error per FP; but then FP varies widely according to the type of computer language, etc., used. For Cobol, if memory serves, the figures typically work out to 105 LOC per FP. That would give us .71 error per 100 LOC, or close enough for conscience to the one bug per 100 LOC that Yourdon cites above. But of course Cobol is one of those wordy (like moi), inefficient second-generation languages, whereas many recent (later generation) languages are much more efficient, with reportedly as low as 15 LOC per FP (i.e., far fewer LOC needed to accomplish the same task). Now, I haven't a clue as to what most SAP programs are written in, but I rather suspect they are fairly advanced languages--let's say with an average of 33 LOC per FP. (Again, feel free to chip in here with detailed info.) You see, with Jones's "average" stat, and translating FP to LOC, SAP would be guilty (and deserving of eternal perdition) of delivering 2.25 bugs per 100 LOC. I find that rather hard to believe. No, I like to think that the good folks at SAP have gotten their development software error rate much lower than that, by a factor of ten or a hundred times, anyway, if only by dint of long practice. And if that is the case, it would throw your mathematical calculations off considerably. Now, instead of an implementation error rate that is 7 times smaller than the *potential* rollover Y2K error rate, we'd be talking about implementation error rates perhaps 70 or even 700 times lower than the potential rollover error rate. Goodness. Or, rather, badness, if you're thinking about January.

Really, at this point, I'm not so much raising criticisms as questions: Just how bug-free (or bug-laden) are new SAP (and comparable) software programs? What can you tell us, in documented detail, about the metrics pertaining to these programs? Have the developers at SAP done a "good discipline" job, in Yourdon's terms, and gotten down to one bug per 10,000 LOC, say (which would then leave the implementation error rate 1575 times lower than the potential rollover Y2K error rate), or is their software really as buggy as the Florida Everglades on a muggy July night?

One other question before leaving this specific issue: when these new SAP systems are implemented, isn't the process gradual and careful, rather than wham, bam, thank you ma'am? Once upon a time, Mr. Yourdon, noting the problems with new software development and implementation, said that the layman (moi again) might suppose that nothing could ever be gotten to work right; the answer, Yourdon said, was that big corporate IT systems were developed gradually, piece by piece, with much wailing and gnashing of teeth, but with ultimate success because of the slow, steady steps involved. Now, again, I don't know beans about SAP, but I would guess that an SAP consultant such as yourself would advise a company to go slowly, to implement bit by bit, and to have the old systems being replaced on hand as back-up in case the need arose. The problem is, when you compare that situation to Y2K errors next year, you might not be giving sufficient emphasis to the fact that next year there will be no back-up, no support, for systems that crash because of those errors. So, even if the rollover error rate were, by divine intervention, much lower than the current implementation error rate, I'm not entirely sure it would mean much for your argument, because we might be comparing the proverbial apples and oranges, and end up with rotten fruit all around. Or, to put it another way: suppose a trapeze artist falls four times in a week, with a safety net below; and then he falls just once the next week, without a net. Is he better off the second week than the first week?

Granted, all those "contingency plans" for next year, and the fact that, according to Cap Gemini, the vast majority of the Fortune 500 companies are building Y2K "crisis management centers" (perhaps showing that the Y2K problem is not "behind" them), are supposed to be a safety net. One just wonders whether or not the net has some holes in it.

My other concerns include all the supposedly "non-critical" systems not getting much attention either in the public or private sector. I don't know whether any of those systems feed into critical systems or not, but that's another issue on which one would dearly love some hard, reliable data, instead of the usual sounds of silence from "official" spokespersons.

Of course, there might even be quite a few "critical" systems unremediated, even in the Fortune 500, depending on how you read that latest Rubin/Cap Gemini America survey. By the by, I was happy to note, at one point in your debate with Mr. Heller, that you basically equated the categorization scheme "critical" vs. "noncritical" with "needed" vs. "not needed" systems. As you might recall from a thread some weeks ago, that was the very point I was trying to make with regard to the way the Cap Gemini America survey questions were worded, and why I said the Cap Gemini America press release and subsequent "Newsbytes" article might have been correct to talk about 54% of Fortune 500 companies not finishing all their "critical" systems by year's end. (Specifically, 2% will finish 50% or less of such systems; another 16% will finish 51-75%, and another 36% will finish 76-99%; data based on survey of 144 Fortune 500 companies and may have been skewed favorably, since the 356 non-respondents may have chosen not to respond because they didn't have such "good news" to report.) But I will grant you that Doc Rubin could have worded some of his survey questions better and that some respondents might have been a bit confused; at any rate, and for whatever reason, the Cap Gemini America survey data seems considerably darker than that compiled by GartnerGroup, etc.

Like the U.S. Senate Y2K Committee, I also have continued concerns about the significant minority of SMEs not doing much (if anything) about Y2K, about the tremendous range in Y2K preparedness overseas, and about the stray, overlooked embedded system that might actually turn out to (a) have a Y2K problem and (b) be doing something halfway important. But all that is getting off topic; besides, like Scarlett O'Hara, I'll worry about that tomorrow. Or the next day.

I beg your pardon for such a long post; given my lack of IT background, I hesitated for a long time (much of the evening) about writing and posting this piece at all. Then I decided these were important points/questions, regardless of whether I was right or (very possibly) wrong. If the latter, I'm willing to make an occasional ass out of myself; Mark Twain once remarked that he had been a writer for 20 years and an ass for 55. Besides, like the half-blind umpire, a fellow still has to call them as he sees them. But the hope is to gain clarification of certain issues--glasses for those of us who need them. Thanks.

-- Don Florence (dflorence@zianet.com), September 24, 1999.

Ed@lizzardranch

First, let me apologize if I offended you or anyone else with my reference to a "grunt" level effort. I have referred to myself as basically a "grunt" level programmer for so long, that I do not consider it a derogatory term, and it was not meant as such.

I have never said remediation of systems was simple. But as you point out, the most difficult aspect is actually finding all of the date references. My point is that, once found, the actual code to fix these lines is essentially trivial. And I think it was just this fact that led many to the replacement of these systems; especially the older, larger ones.

Whether or not we reach "critical mass" was exactly the point of my post. We quite obviously have not approached any form of "critical mass" as yet; my post was attempting to compare estimates of current error rates with the date rollover.

On ERP, I'm sure there are last minute implementations. But these are truly at the tail-end of the Bell-Curve as far as implementations for Y2k are concerned, and are dwarfed in number by the overwhelming majority that are already implemented. Without a doubt, the bottom began falling out of the SAP market about Jan-Feb of this year; the "bench" at almost every consulting company began filling up, and was overflowing by July-Aug.

Yes, more maintenance is required by the "user"; this is almost the very point. SAP, for example, has moved the actual "user" much closer to the actual processing decisions, something that RAD, CASE, and other initiatives attempted, and failed.

You're correct, that many are not programmers, or technically experienced. And they should not be misled by those that are; at least, without being challenged.

Sysman

Explain to me how errors encountered and fixed in the past 12 months, or even currently, add "fuel to the fire" for the rollover. These errors are not cumulative; in fact, after resolution, they lead to a much more stable system.

Don Florence

Thanks for taking the time to review the post. Most, it seems, skim through it at best.

On SAP, and ERP. Yes, the 50% was a guesstimate. In my opinion, on the low-side. SAP began in overseas companies, most notably in petroleum companies, such as Mobil's overseas installations. The numbers quoted relate to R/3 implementations; in 1994, most were doing R/2 (mainframe) implementations. I believe R/3 sales peaked around 1997-1998; which, given an average of 12-18 months for implementation, puts 1998-1999 as the peak for actual implementations. As for extent of usage, my experience has been that a typical implementation usually covers 2-3 modules, which can roughly be equated to at least 2-3 legacy systems. As well, SAP only accounts for approximately half of the ERP market.

As you work your way down to SME's, the use of packaged as opposed to custom software becomes much more prevalent. Even in large coporations, the use of packaged software has been prevalent at certain levels, such as manufacturing plants, etc. The customized systems tend to be more of the corporate, head office type.

If you or anyone else has some harder information, I'd be glad to include it. I have not seen any. As it is, I believe this to be a low estimate.

On the Capers Jones estimate of 5 errors per function point in new development, although not stated, again I believe this to be a low number. Jones is talking about the average software defects in the installed base of software. The inclusion of the "Bad Fix" category in his table is a clue; i.e., the 5 error estimate comes from software that has been installed and to which fixes have been made.

More backup comes from Howard Rubin. In this article, he states that "average defect density of code in the U.S. has varied from 4 defects per 1000 lines of code to about 2 per thousand lines of code today". Also, he states "Average defect densities ranged from 2.3 defects per KLOC in the U.S. to as high as 5.13 defects per KLOC in Canada." Which is roughly equivalent to the 5 per function point stated by Jones.

Because this applies to the installed base, I believe the 5 per function point is low as an estimate for new implementations. The installed base includes systems that have been up and running for long periods, with the bugs increasingly discovered and removed. I would guess that new development injects bugs at a much higher rate, which over time is lowered due to shake-down and fixes, leaving an average of 5 per function point.

And because the more visible and serious errors tend to be recognized and fixed first, those errors above the 5 would tend to be even more serious.

In fact, even more backup comes from the paragraph you quote from Ed Yourdon himself, which stated "One of the most depressing statistics about the computer software industry is that its professional practitioners deliver allegedly well-tested software [the "Delivered Potential," in Jones's jargon] to their customers with an average of approximately one defect (often referred to as a 'bug') for every hundred program instructions". Using a standard of 1000 LOC/Function Point, leaves an estimate of 10 delivered bugs per function point in new development.

So no, I don't believe I overestimated the bugs in software implementations. In fact, I think I underestimated them.

Yes, projects go through a testing phase to weed out as many errors as possible. It is also because of the nature of that process, that I believe Jones is referring to errors in installed code. Few if any statistics are kept on errors discovered and fixed in this phase. Indeed, I imagine it difficult enough to determine these metrics on the installed base, much less on code prior to going through any testing in a project.

As for SAP, as "packaged" software, it is already developed. Even so, it has errors; a quick review of OSS reveals that fact. Errors in SAP implementations are typically injected through incomplete or incorrect configuration decisions, which when discovered, tend to affect many actual "function points". As well, every implementation I've done required large amounts of "custom" development, for interfaces, custom reports, etc., which again interject errors at a somewhat higher rate, since the developer is forced to first determine the underlying structure of data in SAP, a far from simple task. SAP R/3 is written primarily in an upper-level language called ABAP/4; but because of the nature of SAP, with configuration tables controlling virtually all processing, tables which must be accessed and evaluated at each function point, along with the actual processing code itself, the drop in LOC per FP usually seen using higher level languages is not present.

SAP tends to be implemented "module by module", if that's what you're implying by the gradual implementation. But each "module" tends to equate to replacement of at least one full legacy "system", such as G/L, A/R, Purchasing, etc.

Finally, regarding the Cap Gemini survey, I ran across another "clue", if you will. If you recall, the questions on the actual survey read:

What percent of all systems will be compliant and tested by 1/1/99?

What percent of systems do not need to be compliant by Year 2000

Of those that should be compliant, what percent of your systems will not be compliant by Year 2000?

Note that the only question referencing 1/1/99 was regarding all systems.

Now, review this Press Release from Cap Gemini on the survey from August, 1998. It states:

With 500 days to go, only 15 percent expect that more than three quarters of their "critical systems" will be "completely tested and compliant" by January 1, 1999.

Here, Cap Gemini is definitely applying the term "critical systems" to the results of the question regarding 1/1/99, a question that refers to "all" systems.

-- Hoffmeister (hoff_meister@my-deja.com), September 24, 1999.

Thanks, Hoffmeister, for your thoughtful and detailed response.

Re Cap Gemini: Yes, linguistic clarity or consistency doesn't seem to be Dr. Rubin's strong suit (assuming he wrote the survey questions and authored each report). Still, the two relevant survey questions for 1/1/2000 readiness are:

"What percent of systems do not need to be compliant by Year 2000?"

"Of those that should be compliant, what percent of your systems will not be compliant by Year 2000?"

Now, if I were responding to that survey, I'd understand what was being asked: how many systems do I actually need to have done by next year, and, of those, how many will (or won't) be compliant? But I'll grant again that some respondents *might* have been confused, especially since Rubin (or whoever) used "should" (instead of "need") in the second question. Still, I think most respondents probably "got it," taking the two questions together; and if you grant, as you seemingly did in your debate with Mr. Heller, that "needed" systems are "critical" systems, and that "not needed" systems are "not critical" systems, then Cap Gemini was evidently justified (or mostly justified!) in presenting its August stats in terms of "critical" systems. It's just frustrating that such a major consulting outfit couldn't write survey questions with more clarity and detail.

Re Jones's figures: are you sure that his 5 errors per FP refers to *installed* code? At the beginning of your post you sound certain, but partway through you write, "I believe Jones is referring to errors in installed code." Belief is good, but certainty is better. Mr. Jones provides his email address at the start of the article we are using; would you mind emailing him and confirming this point? In fact (and I know this is a lot to ask), would you mind emailing him a copy of your whole "current implementation error rate" vs. "Y2K rollover error rate" argument and see what he says? You and I agree that he is the world's top authority on software metrics; it would be interesting to have his analysis of your detailed argument--and also to know just where he stands on Y2K right now. I did an internet search a week or so ago, to try to find any recent Jones data and/or statements on Y2K, because I was disturbed at not having heard anything publicly from him in months, one way or another. I couldn't find anything on the internet that was recent. As a software consultant yourself, you could communicate with Mr. Jones on a professional level.

As confessed in the post above, none of this is my field, so maybe I'm just having basic conceptualization problems here. But it seems to me that Jones's Table 1 column entitled "delivered potential" is showing the percentage (85%) of errors detected and removed BEFORE delivery of product to customer--that is, before installation on-site. I guess it's that word "delivered" that has me confused. Maybe in this case "delivered" means "after on-site installation and a long time of running the software and weeding out yet further bugs" (though even then you seem to think that the error rate would be 5, not .75, bugs per FP, if I'm understanding your post).

You cite Rubin's article, which, according to you, notes an average defect density of 2-4 per KLOC in the U.S. Let's split the difference and call it 3 errors per KLOC. But this doesn't match the Jones number at all--if you are taking the correct Jones number, that is. According to the Jones article, Cobol typically has 105 LOC per FP (and, of course, later generation languages normally have considerably fewer LOC per FP.) Let's just round that to 100 LOC per FP for Cobol, for ease of calculation. Now, Rubin is talking about bug rates per KLOC, or per 1,000 LOC--and gives an average of 3 bugs per KLOC. If we translated the Jones "5 errors per FP" (for Cobol) stat into errors per KLOC, we would have a whopping 50 errors per KLOC--not the 3 errors per KLOC that Rubin claims. That's why I'm having trouble with your numbers, or, more accurately, with the way you're using Jones's numbers.

Also, if the Jones "5 errors per FP" stat really applied to *installed* code, which you describe as having been already running a "long time," wouldn't the defect removal efficiency number (in the last column of Table 1) gradually approach 0 error per FP, not .75 error per FP, as its limit? Or, put another way, just what do you mean by "long time" here, and wouldn't Jones's stats, to have any meaning at all, have to specify how long the installed code had been running? (If we are to take your interpetation of his figures, that is?) Again, perhaps in my confusion I'm not expressing this well, but I'm sure you get the idea.

Also, since 5 errors per FP would translate into roughly 5 errors per 100 LOC (again, using Cobol as our sample language), or 50 errors per KLOC, I'm left wondering just how any installed program could have been running a "long time," or much of any time, with 50 errors in every KLOC. I know zilch about programming, but my impression is that would be just too buggy to run at all (even conceding a large number of minor, "nuisance" errors). Again, that's another reason that I'm struggling with your interpretation and data here.

Incidentally, thanks for telling me what language R/3 SAP is written in. You note that because of all the configuration tables controlling processing, this language, ABAP/4, probably isn't, in practice, much more efficient (i.e., condensed) than is a second-generation language like Cobol. That makes it easier to talk about the relevant software metrics and error rates (though it would be nice to have a firmer number for the LOC to FP conversion ratio in ABAP/4).

A few final questions. You acknowledge that SAP, as "packaged" software, is already developed. Wouldn't the residual error rate be quite low in a longtime, major, developed software package by a giant, well-respected software company? So wouldn't the installed base error rate be low, too? Yes, I can sense the many difficulties of configuration and customization--indeed, thanks for your explanation on that point, for previously I had no idea that it typically takes 12-18 months to install SAP at a company, though in the post above I did describe the task as "arduous." (My simultaneous condolences to, and admiration for, you.) And I can certainly believe that a significant configuration error would simultaneously affect many FPs. But wouldn't such a major configuration error be occasional, rather than common, and something rather quickly obvious, and hence something to which correction skills could be applied directly and immediately? (Though I've no idea just how long that correction might take!) And in the meantime, doesn't the company still have its old (back-up) systems to rely on? (This gets back to my point in my post above about the possible dangers of comparing apples and oranges, with regard to comparing current implementation errors to rollover Y2K errors and their potential effects.) Plus, you acknowledge that SAP is implemented gradually, one module (legacy system) at a time--which should help to retain the basic corporate safety net during the implementation process. I won't deny for a moment that SAP implementation is messy and fraught with heartache and ulcers, especially since I now know about that average 12-18 month implementation time stat. It's just that potential Y2K rollover errors (however many they might be, relatively speaking) intrinsically seem to me a somewhat different species of beast.

By the way, does SAP (the company itself) keep any sort of detailed stats on implementation error rates, typical configuration problems, MTTR, etc., for its products? Are there any other SAP consultants lurking out there who would like to weigh in on these and other issues?

You note that SAP implementation in a company typically involves all together 2-3 modules or legacy systems. Again, I'm going to show my ignorance of corporate IT systems: just how many key legacy systems does a "typical" big company (whatever that is!) have? Are there any "average" numbers available for the Fortune 500, say? I'm trying to get some kind of idea of the "scope" of SAP in a large corporation: what can be replaced and what can't be.

You write, "As you work your way down to SMEs, the use of packaged, as opposed to custom software, becomes much more prevalent." Since SAP is packaged software, I'm still wondering about just how many Fortune 500 companies are converting to it, and how extensively in the case of each company (see above). From your post, I got the general impression that R/2 (mainframe) SAP implementations have not been all that common in recent years (and of course time ran out quite a few months ago to begin SAP conversions in large companies). Maybe I've read one post or WRP too many by your friend and mine, Cory Hamasaki, but as a nontechnical outsider who hasn't a clue as to which "expert" du jour is right, I do get the occasional willies about just what is *really* going on with the *really* big, enterprisewide mainframe systems. I had been feeling better until that August Cap Gemini America report came along, blast it. Maybe the big corporate IT systems really are in good or even great shape for Y2K, overall, but right now I need more detailed info before being able to accept that premise. Do you think that SAP has publicly available record archives on the number and extent of its product implementations in the Fortune 500?

Again, thanks much for your time and consideration. If I've wasted your time with any points, or raised silly question, it was done out of ignorance, and a struggle for clarification, not out of malice. I try to read as much as possible on this issue from all sides, so long as the reports and posts seem reasonably informed. And whether you prove right, or Mr. Yourdon proves right, or somebody else proves right, I want to thank you, along with assorted other dedicated, thoughtful posters on this forum, for your time and effort. I will be out of town for the next 5-6 days, but I'll catch any and all replies from you and others in the archives when I get back. Best wishes.

-- Don Florence (dflorence@zianet.com), September 24, 1999.