This is hillarious - LOL LOL LOL.

"Embedded Chips Almost Drown Non-Compliant Brewmeister

Copyright 1998 Thomas W. Chittum

Will starving cannibals with AK47s overrun Western civilization? Will your toaster go bananas and chase you around your house going Beep Beep Beep?

I was a computer programmer for years, and after listening to the Y2Kers I am not impressed.

Now listen up and I'll explain in clear, simple English why Y2K is not (with one possible exception) going to send us straight back to Neanderthal times on a one-way ticket.

I will illustrate my points with the tale of Otto Oktoberfest, a brewmeister who failed to bring his automated brewery up to Y2K compliance, but first a little background.

The Y2K bug comes in one form only. When dates from both sides of the millennium are processed in noncompliant systems, gibberish results, often causing the computer to "crash" - simply stop working altogether. Beep! Does Not Compute! I'm Outta Here!

Now, about those pesky "imbedded chips." The Y2Kers never-but-never mention some important facts shrinking their imbedded 900 gorilla down to the size of the Taco Bell Chihuahua.

Some of these imbedded chips are noncompliant, but not as many as the Y2Kers would have us believe. Noncompliant chips have not been produced for some time, and those in place are rapidly being replaced by new, compliant ones.

What about the remaining noncompliant chips that have not been replaced and still on-line on the night of the mother of all meter rolls? They will cause a great deal of difficulty, to be sure, but not nearly as much as the Y2Kers would have you believe. Why? The answer lies in the very nature of the systems that most of these embedded chips reside in.

Broadly speaking, these embedded chips are found in what engineers call "process control systems." The processes these chips regulate are found in many and diverse sorts of automated systems, typically computer-controlled machines in automated factories.

To begin with, many of these time-dependent systems employing embedded chips do not concern themselves with dates at all. They don't know what century it is and they couldn't care less. For example, when these systems are activated to perform their function they start a timing clock already set at zero and continue theirassigned process for a fixed period of time and then stop, reset to zero, and await the next processing cycle. Dates, smates, Don't know don't care.

And one last quite important point about process control systems that do handle dates. It must be kept in mind that the Y2K bug strikes in noncompliant systems only when dates on both sides of the millennium are processed. Almost without exception the systems that do process dates process them only within a limited time span.

Let me give you a specific example. Our aforementioned hero, Otto Oktoberfest has an automated brewery that runs 24 hours a day, seven days a week. He basically pours raw ingredients in one end and automated systems load a conveyor belt with six packs at the other end. Now I have no idea how long it takes to brew beer, automated or otherwise, but for sake of explanation let's say it takes one week, start to finish.

What will happen to a noncompliant brewery as the dreaded millennium approaches? In the last week of December, 1999, the system will start to encounter dates from the next century and fail. The conveyor belt will be told (in effect) that it will be receiving six packs on January 1, 1900 from beer brewed (a century later) on December 24, 1999. Gibberish! Das Chaos! Alcoholic Armageddon! Conveyor belts will spray six packs all over. Brewing vats will overflow, and the hapless Otto will be swimming around in a lake of lager yelling "Mien Gott!"

Is there any hope for our poor, noncompliant brewmeister, Heir Oktoberfest, or is he doomed to slosh eternally in six foot of suds as his automated vats overflow forevermore?

On January 7, 2000, previous-century dates will no longer be feed into the system. All dates being processed will once again be of the same century. The conveyor belt will be told it will be receiving six-packs on January 14, 1900 from ingredients being poured in on January 1, 1900. Wonderbar! Order is restored, and (after prodigious cleaning up) our soaked, but relieved brewmeister will go looking for the dumkopfen programmer who neglected to bring his system up to compliance.

Remember, noncompliant systems work just fine when processing dates from only ONE century, and fail only when processing dates from BOTH centuries. Even as your read this article noncompliant systems processing only dates from this century are working just fine. In the next century, as soon as dates from this century are flushed out, the noncompliant systems will undergo an automatic reset like our noncompliant brewery, and will work just fine once again.

And if you want to get picky about it, even if our brewmeister does not bring his system into compliance after his Y2K swim, his automated brewery will cause him no problems at all and will work just fine for the rest of his life. Unfortunately, his great, great grandson will likewise get a beer bath in late December 2099 when the still noncompliant brewery starts receiving dates from the January 1, 2100.

Bottom line? With respect to the embedded chips, the Y2Kers have been treed by a Chihuahua, and insist that everyone else climb up with them. Thanks but no thanks; I'll keep my feet on the ground.

What about the "one exception" where the Y2Kers may have gotten it right? That's the nuclear power plants, which just might turn into a 9 trillion ton gorilla, and I'll consider them in a later column."

Link at

http://members.aol.com/chittum/y2kbrw.html

-- Andy (2000EOD@prodigy.net), December 22, 1998

Answers

Although I dislike the "nothing to worry about" tone of the article, this is also the conclusion I've come to. In *most* cases, Y2K for embedded systems will either be a non-event, or cause a glitch, or as a last resort be repaired by setting the clock back.

What is omitted are the exceptions (and the one he's mentioned is wrong). There are systems out there (often based around embedded PCs) with realtime clocks faulty at the hardware level. There are systems where the Y2K glitch will cause major damage to the process plant (like the infamous aluminium smelters that self-destructed when the 366th day of their first leap-year arrived). There are plants where fix-on-failure may be disastrously too late, such as those making pharmaceuticals that keep people alive, and of course those in the power grids (electrical, gas and oil). Nuclear plants are designed to fail safe, and I'm pretty sure they will (fail, safely). Getting them restarted will be a nightmare though, especially if no work was done in advance, and in the meantime that may be half your electricity generation capacity off-line.

So, sitting back and not worrying is folly. Checking everything, starting with the mission-critical systems, is what has to be done. Fix-on-failure is an option only if you have established a workable contingency plan for failure. This is expensive, time-consuming, and mistakes will be made, but at least it'll reduce the impact of Y2K.

By the way, the fictional brewery will need extensive cleaning and probably repair of beer-flood damage. The embedded systems may be OK by 7th Jan, I suspect the brewery won't be brewing for quite a bit longer. If its competitors have their act better prepared, it may lose most of its customers and be out of business for keeps by 2001.

-- Nigel Arnot (nra@maxwell.ph.kcl.ac.uk), December 22, 1998.

I don't think even GN is claiming more than a very small % failure rate with embedded chips.

-- Richard Dale (rdale@figroup.co.uk), December 22, 1998.

Embedded chips are truly the "wildcard" of Y2K. On the one hand, only a small percentage are expected to fail or malfunction (1%-7% overall I believe), but unfortunately Nobody Knows where they are and how they are being used. In in the spirit of the season, think of it like a Christmas tree with a string of cheap bulbs -- one goes, they all go. (And yes, like Nigel, I too am of the opinion that the nuclear plants are actually reasonably safe due to their "innards" largely being old analog technology, which makes this article even more funny.)

BTW, I am aware that there are "purists" out there that cringe when they see the terminology embedded chips, prefering instead to see embedded systems, for which microchips are a subset. But I think that the term "embedded chips" describes a valid Y2K problem that lay people like me can understand better.

And since there is not enough time to locate and check each and every chip -- even on so-called "mission critical" systems -- then we are truly going to be entering January 1, 2000 "riding on a smile and a shoeshine".

-- Jack (jsprat@eld.net), December 22, 1998.

Andy,

I think your programming career has been very limited in scope. I think that the explantion you gave for systems which would process only within the calendar year are very few and far between. Yes, there are some, but very few.

I have been working on the Y2K project for 2 years now, and have found that most systems roll from one year to the next, looking forward and backward in time.

I do not know enough about the embedded chips/systems to comment. But, I feel that I want to be prepared for possible problems and continue to search for the good news. I also feel that if there were no problems with these embedded chips/systems that the utility companies would not be telling their customers to be prepared for outages...

-- Carlie (carlie_scott@yahoo.com), December 22, 1998.

Thanks for quoting me, I'm flattered, easily so.

-- William Chittum (chittum@aol.com), December 22, 1998.

It's also very a sad and myopic view of the overall situation. I am a long-time lurker here - not a poster before. I have spent over 20 years in the computer industry - much of that time writing programs in various control languages which eventually found its way into componets called a "PROM" (for Programmable Read-Only Memory).

Many of the programs were written using something called a "Julian date", which used not six digits, but only five. The first two digits were for the year, the next three for the day number within that year. For example, in just a few days we will see a new year. That in Julian terminology will be 99001. There are also complex algorhythms we used to determine the day of the week. If it were a weekday, the control computer was to react differently than if it were a weekend. The control sub-modules (pervasive in many of our applications) tracked maintenance periods by using the current date and subtracting the previous date. Often averages were obtained over long periods of time by using the Julian date and voltages obtained from devices called "transducers" (a special electronic gadget that changes temperatures, pressures, flow rates, etc.) into a corresponding voltage. We also wrote protection routines into the programming. If the control module sensed a radical departure from a long-term average, and that change took place over a very short period of time (as will be the case on Julian 00001), the miscalculation will result in the program defaulting through to either an error routine or one which was written with the assumption that something drastic had malfunctioned. The most common step for us as programmers to take would be to have the control computer generate an alarm condition somewhere and most often "fail safe", which is to say shut down immediately in the mode that would prevent any further "damage". Even though things may be operating normally, the system would most likely close control valves or stop the machinery from operating, having believed that a serious malfunction had occurred. In the event of a sudden and drastic change in operating condition, the control software was written to avail itself of a default backup. This is to say that in the event of a perceived massive failure mode, the control computer would then assume that the condition was real and would cause a backup control computer to be "consulted" for readings. If both corresponded with each other (and they would because the software in each is identical), the condition would assumed to be real, and the outcome would be unpredictable. Due to the proverbial "low-bid" constraints, only one layer of redundancies existed in such systems - just as a parachutist carries but a single reserve chute, or we in our cars carry but a single spare tire. No provisions existed for an identially occurring flaw in both control subsystems.

Even if the date routines were not used in the control computers, they can still cause trouble. For example, in more lay terms, if for some reason you never ever had to put your car in reverse, and yet the reverse gears developed problems which caused metal pieces to be shed internally in the transmission, it would cause problems in the operation of the transmission. Even if the date is not used in the actual operation of the control computer, the division by zero that will occur on Julian 00001 (using the year 00), will cause an undefined mathematical operation within the central processor resulting in the control module halting in an error mode. Downline from the control module, if readings are no longer being obtained from the module (or its backup), results are totally unpredictable.

Summary: I'm one of the countless engineers and programmers who wrote software for these "embedded systems". From over twenty years in this industry, I will say that the problems are both real and unpredictable as it all depends on how the internal ROM software was written -- and there were no standards then. We all did it the way we thought was best and within the very limited memory limits of the ROM.

For the naysayers take it from one who has been and is still in the embedded processor industry. The threat is real and the magnitude unknown. My contact with the industry has been within the confines of my employer -- one which supplies control modules to many, many others.

Thank you for your time.

-- programmer (programmer@embedded.com), December 22, 1998.

Thanks programmer, could there have been a way of coding for 00001 rollover easily (within the constraints you were under).

-- Richard Dale (rdale@figroup.co.uk), December 22, 1998.

Carlie,

Andy was pasting an article from another source.

Chittum is one of the obnoxious one who defy reason in their attempts to be pollyannas in spite of every mounting evidence.

Andy pasted an article from his website. Don't take Chittum's ignorance out on Andy, please. I almost agree with Andy.....it would be laughable, if only it weren't so dangerous.

-- rocky (rknolls@hotmail.com), December 22, 1998.

Happens Chittum is right for a lot of systems - not all. Have a side bet going with a friend that some elements of the power grid will be down exactly 24 hrs - some controllers use two date loops for timing and will have to wait for both to roll over.

Also, in my experience, the ladder program that runs industrial quality PLC's is tested by running the PLC from a laptop as a slave - that is to say that the controls are forced to take on the states the laptop is requiring - the PLC is not in control. Have actually run a large coal tipple on the computer instead of the PLC panel due to a circuit element being burnt out on the panel. Wasn't sure what was wrong - the boss said "make it work" so I flipped the switch and let the problem wait for another time. Plant ran all day with the control panel just sitting there. Replaced the board next day - no big problem. Some plants will no doubt do the same thing come the end of next year.

Don't know about the UK, but over here imbedded PC's as controllers are pretty scarce. Generally, when used as controllers, they are not hidden but sitting in their own case and hook into the control elements via RS-232 or RS-465 or even RS-435. This makes upgrading easy.

-- Paul Davis (davisp1953@yahoo.com), December 22, 1998.

BTW - if you say embedded chips you might mean any chip that was part of the system. If you say embedded system you might mean any part of the system rather than just the control elements. If you want to talk about embedded control elements the correct term would be EMBEDDED CONTROLLER.

-- Paul Davis (davisp1953@yahoo.com), December 22, 1998.

All this complicated talk about chips. After 2000 the only chips of value will be cow chips for gardens i have about 50 tons of them behind the barn and my cows are making more every day. With all of this shit shoveling here it smells like my pile. One says this another says that. Who is telling the truth? My truth is bigger than your truth !! Cow chips--computer chips--imbedded chips--noncompliant chips--potato chips all these chips. Lets all go fishing i will bring the worms another benefit of the pile.

-- Bubba (badhabbit@water.com), December 22, 1998.

So, here we are friends, once again with two completely opposite accounts from the people who work on the embedded chips, uh, I mean systems, oh gosh, controllers, ahh, whatever. programmer and Paul Davis, the former with a very scary testimonial, the latter with the usual happy-face "can do, no-problem" assurances.

Nobody knows, for sure, what Year 2000 will bring, but personally I sure am preparing for the worst!

-- Jack (jsprat@eld.net), December 22, 1998.

programmer said "Even if the date is not used in the actual operation of the control computer, the division by zero that will occur on Julian 00001 (using the year 00), will cause an undefined mathematical operation within the central processor resulting in the control module halting in an error mode."

Now, if the date is not used in the operation of the controller, then why would a divide by zero error occur? And why would a program divide anything using the year digits in the first place?

I don't know about happy face "can do, no problem assurances" Jack, but I do know that usually what Paul Davis says makes sense.

-- Buddy (DC) (buddy@bellatlantic.net), December 22, 1998.

Buddy: Paul always makes sense, but I almost always disagree with his conclusions due to what I fear is his continual "can't see the forest because of the trees" approach to the Y2K problem. And I sure am not going take unnecessary risks! I think that it has been well established that the embedded [chips | systems | controllers] problem is one that is not really well understood, even by those who may have had a reasonable amount of "hands on" experience with them, due to the way they have been programmed (like with "pure" software systems): they pretty much left it up to the programmers to get the job done any way that they saw fit.

On or about 1/1/2000, will the code inside a non-Y2K compliant chip fail outright; branch to an address that was never intended to hold instructions, start executing, then maybe divide by zero; etc., etc.? The point is that Nobody Knows, and these chips are everywhere. Anyone who thinks that a small percentage of bad chips will, for sure, mean a small percentage of problems does not appreciate the concept of cascading failures such as was experienced in San Francisco a couple of weeks ago. Could such things happen due to non-compliant chips? I surely don't know, but there is sufficient evidence on the part of those who do to at least make preparation for such events to be prudent. (And, BTW, I would certainly think it plausible that the power grid could go down, and stay down, due to faulty embedded chips/systems/controllers and their cascading effects.)

-- Jack (jsprat@eld.net), December 22, 1998.

Jack........ what is "plausible"? 10, 20, 40, 70%? Do you mean "within the realm of possibility" or "I could easily see this happening" ?

"And stay down."....? Stop scarin' the women........eek.

-- Lisa (oh@no.com), December 22, 1998.

This is from DATAMATION : 5 billion chips to find out of 70 billion...

". . . "What's the point in spending millions of dollars fixing your computer systems, if you don't have phones, elevators, or heat come January 1, 2000?" asks Michael Harden, president and CEO of Century Technology Services, a consulting company and vendor of Y2K remediation services in McLean, Va.

"This is a global Easter egg hunt, and you don't know how many eggs are out there so you'll never know if you've found all of them," says Brian Kishline, manager of systems engineering for Data Dimensions, a software vendor and consulting company in Bellevue, Wash.

Once you do sniff out these rotten eggs, the dubious payoff is the cumbersome process of contacting the vendor--if it's still in business--to determine the Y2K compliance status of the device (See "Caught in the Y2K time crunch? Compliance databases can help"). Then, if the device is not Y2K compliant, it's generally a matter of replacing or retiring the device altogether. Fixing the code is generally not an option, since usually you won't have access to the source code. With less than 375 days to go, time is too short for most companies to contemplate code repair for the myriad of distributed devices containing embedded systems at their premises.

The scope of the problem

It's easy to ignore embedded systems, since they are necessarily hidden from view (see "On the lookout: a 14-step methodology"). But, hard as it may be to fathom, experts say the Y2K problems inherent in embedded systems are much broader in scope--and potentially much more expensive to fix--than those of business computer systems. Like the Y2K issue in general, the embedded systems piece of the problem is fraught with uncertainty. No one can say for sure how many embedded systems are out there and how many will fail come 2000. Since it's impossible to determine the scope of the embedded systems problem, it's likewise impossible to specify how much it will cost U.S. businesses to fix the problem. From interviews with the top logic chip manufacturers, Harden estimates that approximately 5 billion of the 70 billion chips produced since 1972 are subject to Y2K problems. "The question is, how do you go through the 70 billion to find the 5 billion that will have a problem? It's the quintessential needle in a haystack," he says.

Andrew Bochman of the Aberdeen Group thinks the problem rate is much higher than Harden puts it. "From what my clients are saying, I'm looking at a trouble rate of about 20%" of all devices containing embedded systems, says Bochman, senior analyst for Year 2000 services at Aberdeen, a Boston-based IT consulting company. And Aberdeen's manufacturing clients report spending three to four times as much on their embedded systems remediation efforts compared with their computer systems. Whatever that figure might be will only be determined in hindsight, but whatever the amount is, it's a lot of money."

And remember, the USA is far far ahead of the game than the rest of the computerised world.

Think about it.

And thanks Mr. Chittum, haven't had such a good laugh for a long time. What did you used to program BTW, your VCR? LOL. I crack myself up sometimes.

Prost!

-- Andy (2000EOD@prodigy.net), December 22, 1998.

Lisa, I mean: well within the realm of possibility. Remember, you cannot approach Y2K problems and focus on just one. You have to consider the effects of other potential Y2K problems that are occurring simultaneously, in parallel, which will inhibit fixing a particular one (e.g., power). One simple-minded scenario: the power grid fails because it turns out that a significant number of power generating plants that supply to the grid use Component X, which is not Y2K compliant and fails. Each plant now tries to order a replacement Y2K compliant Component X, but the sudden surge in demand far outweighs the supply, so it might take a month under normal circumstances to fill the orders. But, meanwhile, without power, nobody can fill anything, etc., etc.

There are many who will look at the above and say that it is completely far-fetched. Others (a la Infomagic) might say that, if anything, its way too optimistic, since it implies that there is a working transportation system, clean water, food, communications, etc., that would also be there for that month.

Nobody knows....

-- Jack (jsprat@eld.net), December 22, 1998.

I'll second programmer's observations of embedded system controllers having problems with a "divide by zero" come 1/1/00. It was the "autopsy result" on the computers in the first Y2K date rollover test I was associated with. Things haven't gotten much better since then (October of 1987).

After twenty-five years on the hardware side of the equation, three spent working with PLCs, I can state that there are some companies still selling non-Y2K capable equipment. And there are non-Y2K compliant programs being loaded and shipped to customers this very day. If I can say anything good, it's that I know these things are only going into manufacturing equipment and not something critical, say power systems.

WW

-- wildweasel (vtmldm@epix.net), December 22, 1998.

As a programmer who works on completely different kinds of systems, let me just quote our common refrain:

"It oughtta work. There's no reason why it wouldn't work."
...TEST...
"Oops. What do ya know, it don't work."

In code, the Law of Unintended Consequences is king.

-- Shimrod (shimrod@lycosmail.com), December 22, 1998.

Shimrod: Amen brother. If I had a nickle for every time I incorrectly diagnosed a system problem and said "that's got to be it", well, I'd have a lot of nickles.

-- a (a@a.a), December 22, 1998.

Jack - not happy-facing just stating the size of the problem as I see it. Remember the poem about the blind men and the elephant.

About ladder programming - a neat graphic way to program industrial processes. Give it a try -- go to this link

http://home1.pacific.net.sg/~triangle/trilogi.htm

Have fun!

You might also note that while this program is for logic ladders for three families of PLC's, I first encountered it on Allen Bradley PLCs - which are pretty common in industry. Takes about a week of classes for a beginner to get good enough to write simple but useful programs.

And BTW - Allen Bradley ne Royce state flatly that all current models and all models for quite a ways back are Y2K compliant. The ones that aren't are made to be programmed from 286's - have seen just one of those that still worked in the last 3 years and wouldn't have cared if it quit - a piece of junk.

-- Paul Davis (davisp1953@yahoo.com), December 22, 1998.