As we enter the home stretch, what have we learned and what can we expect? Simple questions, huh. Discussion has on many fronts become needlessly polarized. Skeptics, as Yardeni now refers to himself, give elaborate well-contructed, intellectual arguments which if accepted lead to the conclusion that as a people we have shown depths of stupidity which Kurt Vonnegaut could never have dreamed up. Optimists, seem often concerned with attacking the position that things could be in trouble, not over stated concern for potential suffering and disruption but out of refusal to acknowledge that such an ignorant error could have occured. The 100s of billions spent worldwide attempting to correct this error attest, that yes we are in fact that stupid. At this point I suspect everyone reading this has experienced some form of Y2K or seemingly Y2K induced distruption, I know I have. While the prevalence of these errors has been wonderfully low, they are often of a type where one can easily extrapolate to dire what ifs. This is also the case in more prominent Y2K failures. Accepting that Y2K failures are in fact real and not theoretical, I have in my opinion not seen much truely optimistic news. To be blunt, I would guess that the majority of computers worldwide at risk of failure are unremeditated. I base this off 1) Many countries have adopted in essence a fix on failure policy as have many small business. 2) Those countries and organizations choosing to remediate have been concerned primarily with mission critical systems which often are a small subset of total at risk machines. 3) Many organizations choosing to remediate have still not stated complete compliance of "mission critical" systems 4) Independent Validation and Verification is seemingly performed sparodically and when performed frequently reveals remaining or newly introduced errors incompatible with a critical machine's mission. 5) Interfaces between modified machines remain problematic as brought up by Dale Way and Howard Rubin.

Aside from potentially looking off a societal precipice what does this potential imply. To look into the effects of the Y2K malady, I suspect it is more useful to consider the function of these machines within their societal context rather than some sort of Linnaen or engineering based classification scheme. As an aside, to make this point, I once studied the German philospher Hiedegger, forgot almost everything he said but he was always talking about something's Daseign (sp) or its contextual meaning. For instance if you forget your car keys, the Daseign of your car has suddenly changed to a ton of worthless metal (simillarly if there is no gas ...)

Manifestations of Y2K may be divided into 3 groupings. The first is what I would call bookkeeping operations, these include the partial automation of tasks previously done by pen and paper. Examples might include balance sheets, bank ledgers, pay stubs, property titles etc. The second area I would call process control. Process control involves the automation of labor saving devices. Examples include SCADA systems, PLCs, elevators, industrial cranes, electrical generating and distribution systems. I am more comfortable with this functional type definition, then with the strict embedded system, versus mainframe legacy system. For instance a PC when used for an accounts package would be considered a bookeeping system, the same PC, controlling a series of agricultural sprinklers would be involved in process control. Only in considering the functions of these machines in context can we hazard a guess at the ramifications of failure. The third area of concern is social/governmental response to a known stressor of unknown magnitude, which I won't comment on.

As regards bookeeping, I suspect the wealthy are in good shape at least for a time. I base this off the following observations. (If challenged I'll look up the bookmarks, would take a while right now though). 1) Chase Manhatten recently misplaced 40 Billion dollars (they found most of it). To put this in context this is nearly five times more then the entire US Government has ever spent on Y2K remediation, it is almost half of what the entire country has spent on Y2K, it is more than many countries GDP. At the same time, the stock markets roared upward. I don't know how this happens, but I see a bunch of CEOs punching the chase manhatten CEO on the arm and saying, "so and so, you wild and crazy guy, what kind of prank are you pulling, this is even better than your frat house prank. 2) There is an initative to forgive all third world debt being seriously considered. If all the governmental bookeeping were screwed in the thirdworld the lenders wouldn't bat an eyelash. There appear to be people capable of shrugging off billion dollar hits to the world economy without global problems. Now the 80 year old lady living off SS who needs catartact surgery but can't get Medicaid to work, well there's a problem. Similar type problems have been surfacing with student aid, DMV functions and teacher pay. The time course of bookeeping failures is also different than process control. There would be an expected gradual increase (though the type of curve is not known) with a gradual decrease after 2000 (gradual self-correction as Mr. Way points out). The backside, after 2000 might be expected to be more severe as many groups are utilizing workarounds such as deferring reference to year 2000 dates as was/is reportedly done on some welfare programs. Also a number of currently functional systems slated for replacement msut be replaced before (not after) 2000, if the replacement implementation is problematic it shows up after the cutover. Finally many bookeeping type errors have delayed manifestations. This will all be a problem but it is not what I am most concerned about. People don't eat paper, the government shut down over a budget debate for a while and no one was all that concerned.

Process control is what I am worried about. Two quick points here, I have reviewed all 100 some odd case failures of embedded systems (generally involved in process control) posted on the IEE website. In not one did it state, the clock was forwarded to 12/31 and the system failed, in all those that failed from Y2K noncompliance failure was at roll-over. My pentium 100 did not fail until rollover, and really who gives a hoot if some obscure spreadsheet cell was behaving erratically prior to the roll-over. 2) The installed base of machines involved in process control is many times that of bookeeping. Process control systems are also different from book-keeping operations in that they are often making decisions, ie, close this valve, open this breaker, this logic must be consistent and may not be at rollover. The types of failure with process control could be catastrophic on more than paper. There are only, I suspect, poorly defined methodologies for tracing process control system failures, as these failures would also mimic human failure or natural disaster type problems. As noted, if process control error leads to a chemical spill, fire, or explosion, bug tracing in assembler might be the last of one's concerns. Well, enough of this. To sum up, don't count your chickens before they hatch, its called the Y2K bug for a reason. If there's no problem, e-mail in the new millennium and let's have a picnic. Oh almost forgot, the majority of problems in the electrical industry regarding Y2K strike me as process control systems, (PGW, Con-Ed and others have already had billing prblems, in the big picture who cares). Hopefully, the degree of exposure to Y2K related failures in process control in the electric industry and neccessay ancillary industries is small. I start a research oriented position next week, and probably won't be posting or reading as much here as I'd like. Take Care!

-- Anonymous, November 19, 1999

Answers

Paul,

What failures in the the electrical strike you as process control problems? The ones you cited in the same sentence were related to billing.

As far as stupidity goes, Y2K has been a real problem requiring scrutiny and dilligence in repair/remediation. I would contend that in the electrical arena, the bulk of the work that required immediate attention is done (and all will be completely done by the rollover after a few immanent scheduled unit outages). A debt is owed to those who got the issue raised in the corporate consciousness. True, the problem turned out to be not as large as feared, but still - to have done nothing would have been a truy stupid risk.

-- Anonymous, November 19, 1999

Paul,

Hear, hear!

Well spoken.

-- Anonymous, November 19, 1999

Paul, I find the following paragraph of yours quite impressive in its logical grouping of the significantly differing manifestations of the Y2K problem: "Manifestations of Y2K may be divided into 3 groupings. The first is what I would call bookkeeping operations, these include the partial automation of tasks previously done by pen and paper. Examples might include balance sheets, bank ledgers, pay stubs, property titles etc. The second area I would call process control. Process control involves the automation of labor saving devices. Examples include SCADA systems, PLCs, elevators, industrial cranes, electrical generating and distribution systems. I am more comfortable with this functional type definition, then with the strict embedded system, versus mainframe legacy system. For instance a PC when used for an accounts package would be considered a bookeeping system, the same PC, controlling a series of agricultural sprinklers would be involved in process control. Only in considering the functions of these machines in context can we hazard a guess at the ramifications of failure. The third area of concern is social/governmental response to a known stressor of unknown magnitude, which I won't comment on.

In my opinion, you have an excellent grasp of this aspect of Y2K. In my opinion, this grouping is far superior to those of the Gartner group and others who have "created" new terminology to try to group embedded systems. I especially like your approach, since you have picked up on the important aspect of whether a PC is used in equipment control - this is a significant factor in the types of problems that are likely to be encountered, compared to lower level equipment that utilizes only firmware programming in a PROM (together with a real time clock, microprocessor, etc).

Now a comment regarding PCs used in controls - this also varies tremendously. The "typical" use of a PC type computer in a large scale control system is as an interface only, with separate dedicated firmware based control systems actually doing the controlling. If you haven't worked with Distributed control systems, PLCs, etc, but are familiar with PC software programming, you might be likely to assume that a "bug" would be hard to identify in the program, since software programming is in difficult languages such as C, C++, etc. That would be a mistaken assumption. Most control application is at a far higher and easier to understand level - ladder logic programming for PLCs is quite simple to use (although growing more complex by the year'), and also a lot easier to debug than software programming. DCCs are more complicated, but even these use well structured programming blocks that make diagnostics and troubleshooting problems a lot easier than debugging C code.

There are MAJOR differences in date usage in business software and y2k bugs in the firmware based control systems - control systems rarely used the dates for control functions, I would estimate that well over 99.9% use dates as "information", i.e., date stamping. I cannot emphasize this enough, in my opinion, this is the single biggest factor in why Y2k has been demonstrated to not affect "embedded systems" to the degree postulated a few years a go. This has been proven out by the fact that almost all y2k problems in firmware based equipment are minor date stamp/display problems. (There are exceptions, I am aware of several devices that use the dates for calculating the decay rate of radioactive isotopes, but the point is that date calculations and controlling on dates is compariatively rare).

The second biggest difference is that many firmware based control devices don't have date functions at all, whereas all modern PCs have an RTC.

Now on to use of PCs to interface to controllers. In many cases these PCs run software programs and act as a Man Machine Interface (MMI) to the dedicated controllers (DCC, PLCs)for viewing, and "manual" (non-automatic) control by the operator. Most systems that I am familiar with do not affect the dedicated controller if the MMI fails - it continues to operate in automatic, and there are hardwired manual control switches.

Now, having said all of that, let me add that times continue to change, and over the last few years the line between dedicated controllers and PC based control have blurred in some areas of newer designs. We are seeing the era of PC based controls now - not a substantial number yet, but growing. First Unix and other operating systems, now NT based systems. And not just the MMI, there are controller cards that fit in the PC slots. (FYI, do not confuse this with PC based data acquistion I/O cards, these have been around since shortly after introduction of the IBM PC. The difference is now were controlling with them). I wan't to state here that I have not seen PC based controllers used in mission critical applications and haven't seen them used in power systems, but I doubt that this is the case in general industry.

A side note: If people need to fear something, I suggest they fear IT type programmers installing PC based control systems without the proper fail-safe design factors that engineers require, it's probably just a matter of time...

I also agree with you that most of the potential for serious y2k problems is in software applications, primarily financial transaction software.

I think that after the new year, most people will come to see that y2k bugs in embedded systems didn't cause widespread and significant problems as once predicdted, and that Y2k was more of a problem in software based business applications, although it did not a significant impact on society due to remediation. I expect to see y2k bugs in the places we already have - the places with the most bureaucracy and general inefficiencies that impeded the y2k efforts - local, state, and federal governments.

Regards,

-- Anonymous, November 19, 1999

FactFinder says: "A side note: If people need to fear something, I suggest they fear IT type programmers installing PC based control systems without the proper fail-safe design factors that engineers require, it's probably just a matter of time..."

One of my close relatives who works in the aluminum extraction industry has already seen this happen -- eight years ago. The manager of IT in his company convinced the CEO that the chemical process automation work then being done by expensive outsourced engineering talent could just as well be done by in-house programmers in the IT department, through proper choice of technology (PC based of course), plus appropriate training for the IT programmers. After all, "programming is programming".

Three years and $10 meg later, the whole effort failed miserably, and the IT programmers were sent back to doing what they knew how to do. Fortunately for the people working onsite and downwind of these plants, the impact was on corporate profits, as opposed to the safety of employees and the public. However, it took the forced retirements of both the CEO and the CIO before the mistake was admitted and corrected.

-- Anonymous, November 20, 1999

Scott, Great story, thanks! Eight years ago, this may have been one of the first examples. I would guess that it's also happened more times since then.

One reason that I expect this to start happening now is that I am seeing more and more control software and hardware made for running on PCs with NT as the operating system. Thus, the distinction between dedicated types of computer controllers vs. PC based systems once used only as the interface, is starting to blur.

The software programs are more and more Windows/NT based. With the software and control functions both PC based, and using Windows based software, I am afraid we will be seeing more and more trained programmers doing systems programming traditionally done by engineers and technicians. I have already seen this first hand.

Regards,

-- Anonymous, November 20, 1999