The linchpin of the Y2K story is the measurement of "compliance." But this linchpin is based on a scam, namely, that compliance percentages as reported can be the subject of meaningful comparison and analysis or made the basis for confidence in a safe Y2K outcome. They can't be, for these reasons:

FACT:

Mission-critical systems are the sole focus of compliance. The Y2K story is conventionally reported as though the compliance of mission-critical systems is adequate to cover meaningful Y2K compliance. It isn't.

WHY:

Between 60 to 90% of all systems are not being considered mission-critical and never enter an assessment. Even assuming none of these systems are, in fact, mission critical, nearly all of them compute dates. Given the expectation that non-remediated software systems are at significant risk of failure, we can assume that 'x%' of these systems will fail. Yet, no consideration is being given publicly to the impact of the failure of these systems upon the enterprises in which they reside. Worse, although the phrase "mission-critical" is reported commonly by the media, the general public equates "mission-critical compliance" with "enterprise compliance" and assumes falsely that the failure of non mission-critical systems is trivial.

FACT:

There is no logical basis for assigning systems as mission-critical yet this demarcation is the foundation for constructing estimates of compliance.

WHY:

While two decision-makers within an enterprise will converge to 'x' degree in making judgments (ordinarily based on experience) about the criticality of given systems to an enterprise, there are no quantifiabley defensible standards for determining why a particular system is mission-critical.

FACT:

There is no logical basis on which to compare the assignment of mission-critical systems by different enterprises, even within the same industry.

WHY:

See fact above. Different enterprises judge different systems to be mission-critical. There is likely to be 'x greater' degree of convergence within an industry than when comparing across industries, but the convergence will be partial within an industry and still less across industries. Therefore, comparing Y2K compliance percentages across the entire Y2K effort is ridiculous.

FACT:

Even mission-critical systems are being "redefined" out by some enterprises, skewing compliance percentages.

WHY:

Because some enterprises have a PR stake in asserting 100% compliance of mission-critical systems but are behind schedule, the number of mission-critical systems is being defined down progressively over time (cf. the United State government). However, no rational justification is made for the reclassification (that is, this remains a subjective judgment call of the individual with authority to assign systems).

FACT:

There are no standards for determining which tasks should be assigned to which development phase in the life-cycle and then for measuring the actual duration of those tasks.

WHY:

Although there are historically sound percentages assigned to different portions of project life-cycles, only a minority of enterprises manage any life-cycle model at all. Moreover, even life-cycle and metrics advocates believe that different types of projects require the application of a different life-cycle model. There is no single standard model. As a result, it is vanishingly unlikely that enterprises who have never applied a rigorous life cycle (the majority) are accurately assigning tasks to the appropriate life-cycle phase (e.g., assessment, remediation, test, etc) and even more unlikely that accurate statistics on task completion are being collected. Most likely, these "statistics" are being subjectively "estimated" without recourse to data, or, at best, with each estimator providing their own quantifiable criteria for the estimate that is generated.

FACT:

Many organizations are likely to include systems found compliant during assessment within compliance percentages, skewing the schedule for completing overall compliance.

WHY:

Once a system is judged mission-critical and passes through the assessment phase and/or receives trivial remediation, it is conventional to include it within the "compliant" bucket. Since most, though not all, remediation will await completion of assessment, the earlier reports of compliance percentages are probably overly optimistic. Consequently, it is also probable that compliance schedules will lag severely at the end of the cycle.

FACT:

Compliance is self-reported (unaudited) except in rare circumstances. Industry experience demonstrates that self-reported metrics negatively predict outcomes.

WHY:

To produce accurate metrics, each stage of a project's lifecycle must be subjected to external audit (design reviews, code inspections, execution of test cases, etc). A tiny minority of IT organizations utilize external audits. Programmers nearly always over-report completion and under-report problems. This is hidden from view until late in the cycle when the results of self-reporting become visible externally as failed systems.

FACT:

Compliance of software systems is meaningless taken by itself without reference to the compliance of embedded systems. But it is widely acknowledged that there is no useful way to test, let alone measure, the compliance percentage of embedded systems.

WHY:

Because so many embedded systems are inaccessible and because systems of identical type have been shown to fail while others of the same type do not fail, it is impossible to measure their compliance. For that matter, it has proven impossible to develop agreement even on the number of systems at risk. Consequently, to discuss Y2K compliance reduces effectively to discussing the possible compliance of mission-critical systems, leaving out both other software systems and all embedded systems.

CONCLUSIONS:

As stated. The entire compliance game is a sham. I resent the way that many of the so-called Y2K gurus have played along with this, thereby lending it credibility. Yourdon has been one of the few who has kept the heat up about this (Dij` Vu and other articles).

Therefore, I pay strict attention to these facts, not to the compliance percentages:

1. Are self-reported schedules slipping?

2. Do reports of compliance percentages actually go backward (first 90%, then 80%).

3. Are the numbers of mission critical systems downsized (they are never upsized)

4. Are budgets accelerating and at what rate is money being spent? (BTW, we could take an equally critical look at the budget game which neatly parallels in scam-ness the compliancy game).

5. Are enterprises reporting that they will not make it by YE 99? (Just starting, but watch the trickle become a flood).

I also pay serious attention to anecdotal reports although I have to trust my "nose" (hey, I am a dog after all). After 20 years, you can detect the ring of truth in the nuances of someone's narrative.

I've said before (about NERC) that compliance is such a sham that it is possible the Y2K remediation effort is further along than we think. It is extraordinarily unlikely (REPEAT: EXTRAORDINARILY UNLIKELY) given my knowledge of what goes on inside the glass houses but who knows? THERE IS NO WAY TO KNOW.

But it is no small thing to KNOW that compliance reporting is a sham. More specifically:

The scam of the "mission-critical" compliance game + The complete inattention to the vast majority of an enterprise's "other" systems + The inability to measure the exposure of embedded systems = A disaster in-the-making

"The Y2K compliancy game is a scam." It's almost as bit a scam as Y2K itself. No, better: it is an integral part of the scam that gave us Y2K in the first place. Don't ever forget that.

-- BigDog (BigDog@duffer.com), January 22, 1999

Answers

Arc Angel commented:

" Okay, thats my little sarcastic recreation of a shakespearian DGI. So now that we have that cleared up, hand me that rope and I'll get busy stringing up the Senators, Congressmen, and Congresswomen by their lying asses and you can start that funeral pyre for our esteemed members of the Presidential Cabinet... and of course Bubba himself. Oh I can't wait to see the flames of justice... for they burn HOT and BRIGHT! "

We may not have a problem with fossil fuel after all. Should be enough here to provide many years of comfortable heat.

Ray

-- Ray (rayl@whc.net), January 22, 1999.

Re: "burning" at the stake of Y2K compliance. If Y2K < Infomagic, there are going to be "Nuremburg" style inquests into the deceit of compliance. There may well be much grandfathering-protection of pre Y2K IT offenses, but there will be a "never again" attitude about the way software is done.

While there are a few organizations that do a bang-up job (thanks to efforts over the decades by people like Yourdon, Jones and others), the ordinary person would not believe how sloppy the process is.

Once again, what a joke that the same bozos who gave us Y2K are now passing out compliance percentages as though the numbers had semantic content to them. The only use of these numbers is to turn them back on the perpetrators (as Milne and I did with the NERC stuff) to show that the "data" is self-contradicting. You could pretty nearly define a pollyanna as "someone who believes compliance percentages are meaningful." ROTFLMAO.

-- BigDog (BigDog@duffer.com), January 22, 1999.

Superb post. I've been particularly worried at the fact that nobody ever mentions all those supposedly "noncritical" systems. And, after plowing through both the Sept. and Jan. NERC reports, I had many suspicions about the way that survey data and "compliance" percentages were being used. I just lacked the background to formulate those suspicions concretely and logically.

-- Don Florence (dflorence@zianet.com), January 23, 1999.

I am not a computer expert. I am an accountant. We are taught to look at the numbers and the text and evaluate them. Your conclusions are the same as mine. Only it appears that you have the extensive computer knowledge which I lack. Thank you very much. I looked at Yardeni's latest post on his website. He has a table that summarizes the OMB numbers used to report their % of completion. The way they report the numbers, no trend lines can be drawn to project the % complete by 1/1/2000. They keep decreasing the number of mission critical systems. If you change the number of mc systems back to the higher ones reported, they were only about 50% complete in Nov. 98. Their numbers do not mean much. Now tell that to the press.

-- Sue (Conibear@gateway.net), January 28, 1999.

Sue --- we would do better if accountants were analyzing the compliance percentages, seriously.

As I've been ranting (with good reason), the numbers "don't add up." It has nothing to do with being a geek, but with understanding the simple rules by which one thing can be reasonably compared to another. And, as you point out to us yet again, even the pitiful rules keep changing mid-stream (the "mission-critical" game).

I continue to insist that they are so messed that we could be further along than we think (though all the *anecdotal* evidence is against this), but reliance on the percentages for OPTIMISM is ridiculous.

The most we can say is this: someone who says they are 50% compliant is further along than when they said they were 25%. But the verifiable content of that statement is almost nil with respect to predicting positive consequences for Y2K.

-- BigDog (BigDog@duffer.com), January 28, 1999.