Embedded Systems: Do You Really Wonder Why Some Industry Experts Really Don't Get It?

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

From the February 1999 issue of Electronics Design Magazine comes this editorial. This magazine is the electronics industry's business-to-business source of information on new products, technologies and major issues affecting the electronics industry.

The editor is DGI or DWGI as I read this piece. I'll post a few comments after the end. I apoligize in advance for any typos, this won't scan in so I'm going manual here. (snip)

A few readers took me to task for having a little fun with the Y2K problem in one of my editorials, "No Y2K Shelter For Me" (ELECTRONICS DESIGN,Nov.16,1998,p.14). I was told that the impending doom that will befall the planet come 12:01 a.m. on Jan.1,2000 should not be taken lightly by the media. Also, everyone should cancel their parties and hotel reservations for New Year's Eve because the new millennium doesn't officially kick off until Jan.2,2001.

Let's start with the date controversy. I've long known when the supposedly "true" millennium begins. In fact Arthur Clarke, author of 2001: A Space Odyssey, is so miffed by the premature celebration that he's issued a press release to correct the calender-challenged. He explains that because the Western calender starts with Year 1, and not Year 0, the 21st century and third millennium won't begin until Jan.1, 2001. If I were sitting in Clarke's shoes, I'd be miffed too. Imagine how many books he could sell if the millennium were celebrated in 2001.

And who says 2001 is the official date? According to a Jan.18 TIME magazine article, "The End of the World as We Know It," we already missed the millennium. A sixth-century monk (Dennis the Short) based our calender structure on the date of Christ's birth. But historians now place the birth no later than 4 B.C., the year King Herod died. So the third millennium would've started no later than 1997. I'm sure the societies that don't use this calender are laughing at our 2000/2001 controversy.

Look, it's been a tough 1000-year stretch. People can be excused if they want to celebrate a little early. Even Clarke admits there's something a little magical about reaching the number 2000 (all those zeros). Besides, if the apocalyptic cheerleaders are correct, at least Earth's citizens will enjoy one whopping party before all hell breaks loose after the ball drops in Times Square.

This brings me to my previous editorial. Some readers criticized me for not taking the problem as seriously as an impending nuclear attack. **Yes, there will be problems, especially with date-sensitive embedded systems.** (emphasis mine, WW) One worst-case scenario is that some water supplies will truly go down the toilet because the glitch will cause the water companies' computers to dump more chlorine than required. There's the ever-popular Internet buzz about all of the superpowers' ICBMs lighting up the sky. And we're all warned to cash in our IRAs and bonds for extra cash. Don't fly during January and stock up on toilet paper and ammo.

Give me a break. As I said last time, I'm not building a survival shelter like that west-coast computer programmer. First of all, I can't solve the problem. I also don't scare easily. And if problems do occur, I'll do what I can in my community and workplace to work through them. Remember, Y2K is a one-time problem. It may take a while, but it *will* be solved. We'll deal with it. Some think it's actually a blessing in disguise, because billions have been spent to upgrade our computer infrastructure. This will make the U.S. more efficient and competitive in the years ahead.

But what bothers me and should bother you are knuckleheads scaring the heck out of people --especially children-- with end-of-the-world predictions about the millennium/Y2K double whammy. Some religious extremists, stargazers, survivalists, consultants, and "journalists" should be ashamed for fueling panic. Card-carrying New Agers are fanning the flames because the alignment of planets in May 2000 will torch the Earth. Jerry Falwell has a book out entitled, Y2K: A Christian's Guide to the Millennium Bug. Gee, I thought it was a computer problem.

At a recent American Business Press board meeting, chief editors from top-tier business-to-business magazine companies discussed the misinformation being spread about Y2K. To combat it, we're planning a web site. ABP members will post Y2K articles with factual information and sources to quote for articles.

Panic is already spreading. I read about one man stocking up on toilet paper to use as "barter" if the financial institutions collapse. And people are buying dried food and firearms. Great... more guns in the hands of amateurs. Let's hope that all we hear on New Year's Eve are fireworks, not gunshots.

I trust that you'll do what you can to promote a state of calmness and objectivity until this bug is history. (end snip)

WW's comments:

I don't have access to the first editorial, editorials aren't on the magazine's web site. I may have to spend a lunch hour at the company engineering library to dig that one up.

I'd expect a Y2K editorial in an industry trade magazine to be a bit more objective and business-like. I thought I was reading something from TIME. And this is an industry leader? Gun-buying, children-scaring, religious extremist, stargazer, survivalist, consultant, "journalist" and New Ager comments aside it boils down to this one statement.

"Yes, there will be problems, especially with date-sensitive embedded systems."

It's the pearl in this editorial. For all the "polly talk" the writer throws out, this is the embedded systems quote of the millennium. With embedded system applications, the more complex the usage, the more likely the use of dates in the embedded systems operations.

TRANSLATION: Oil refineries and power plants are complex applications. VCRs, microwaves and other home appliances are not. Where should we expect embedded systems Y2K problems and what are the potential impacts?

In every issue of this magazine that's crossed my desk for the last nine months, I've looked for Y2K-related info. It's only been on the editorial pages.

In fact, just a few pages away from this piece, ads were run for embedded application, single board 386 computers. And from what I know, there are no Y2K-compliant or Y2K-ready, 386-based computers.

WW's observation: People high inside the electronics industry still can't get the big picture on Y2K. And less than one year before Y2K hits, companies are still selling Y2K-susceptable components. And these can find their way into new products at this late date. I hope not into products being used in Y2K remediation projects.

I posted this to try and give a perpective on the point-of-view inside the electronics and computer industry. From time-to-time, we've had posts about how Silicon Valley doesn't "Get It" on Y2K. This shows that the hardware engineers are just as myopic as the software engineering crowd.

I look forward to seeing what's on the new web site mentioned in the article. Will it be a balanced site or will JBD be the webmaster? I still wonder about what's given some people the ability to "see the big picture" of how seemingly unrelated systems integrate with each other. And these guys are making the big bucks?


-- Wildweasel (vtmldm@epix.net), February 25, 1999



did you notice his comment about y2k being a 'one time problem'...I take it engineers aren't required to study world history?? I mean there have been some fairly serious 'one time problems' that were 'solved eventually'...yeesh!


-- Arlin H. Adams (ahadams@ix.netcom.com), February 25, 1999.

>In fact, just a few pages away from this piece, ads were run for embedded application, single board 386 computers. And from what I know, there are no Y2K-compliant or Y2K-ready, 386-based computers.

This rather gives the game away. 386 chips don't have realtime clocks on. They are utterly, completely compliant. The embedded systems into which they are assembled may or may not have realtime clocks which may or may not be compliant, and if they aren't it may or may not matter because an unused noncompliant clock is no danger unless there's a design flaw in the silicon itself. It's all been discussed to death many times before in this group. You haven't even bothered reading the archives.

I'd hope that any 386-based embedded system being sold today is compliant, though doubtless there are a few hapless incompetents out there still making duds.

As for the all-or-nothing world view you're promulgating, it's plain wrong. Reality is far more complex, and has coped with large-scale foul-ups before, such as WWII (which was a far bigger mess here in Europe than it was for lucky USA mainland residents). This isn't to say that there's no reason for great concern: there is. Y2K is utterly without precedent which makes for huge uncertainty, so prepare for whatever worst-case makes sense to you even if it's TEOTW. But please, take the trouble to find out what you're talking about before posting authoritative-sounding balderdash!

-- Nigel Arnot (nra@maxwell.ph.kcl.ac.uk), February 26, 1999.


I've been in the electronics industry (military and civilian systems) since 1972. I've been working with embedded systems since 1975. The last two years I've spent the majority of my time rooting out Y2K- noncompliant 386 and 486-based stand-alone microprocessor systems and PLCs. I ain't no newbie to the business, and I am no "Internet know- it-all" who doesn't have a real clue about the about the issue.

386 and 486 processors rely on a support chipset, even if the product is referred to as an x86 device. The 386 and 486 industrial single board computers I've seen do use RTCs in their support chipsets, and I haven't seen a 386 yet that wasn't coupled to a flawed RTC. I've seen some 486 devices that are Y2K ready but most have suffered from the RTC bug also. The Motorola microprocessors I've seen in devices running in Unix environments have not had problems. Same for the DEC Alpha and the Power PC chips I've seen used in some custom embeddeds.

My observation about the 386 processor advertisement is a reaction to not seeing any statement in that ad, or any ad in Electronics Design which comments on Y2K readiness of any product.

Because as late as last summer, I was seeing brand-new, non-compliant 386-based systems being installed in some products, it seems strange that if a company has designed and built a non-Y2K problem suceptable 386 or 486 based device, that they would not seek make the readers of their ads aware of that fact. Instead there were no comments on Y2K readiness of products or applicable ISO standards in the ads, nothing at all.

In my experience, the hardware makers have said Y2K problems are the responsibility of applications designers and programmers. Of course these folks remark that they've gotten no information in product specifications concerning the ability of devices to handle the Y2K transition. Finger-pointing around in a circle doesn't help the problem, but that's been the procedure on two recent projects I've worked on. Rather than fixing things, everybody wants to play CYA.

I've been aware of Y2K failures in embedded systems since 1987 when I was part of a DOD test team which looked for and found what are now called Y2K failures. Twelve years of observing the problem leads me to believe that some people in the electronics industry still don't understand, or care about the problem. I look at this editorial coupled with the ads as glaring examples of an industry that is sending itself mixed messages about embedded devices and Y2K.

If the electronics industry itself hasn't come to grips with the idea that there's a larger scope of concerns which intimately involve their products, then we may not ever really learn how widespread or not the embedded devices issue really is. That is, until we're in the field doing "autopsies" of major system failures trying to figure out what went wrong. Like the Challenger accident, determining that a seemingly insignifigant item caused a catastrophy is no relief from the catastrophy. But determining that such an insignifigant item could cause a catastrophy and eliminiating the possibility by correcting the problem beforehand is a lot more effective and rewarding.

Unfortunately with ten months to go before we see what will happen, I'm afraid after all's said and done I'm going to be doing a lot of field work trying to find out what went wrong.


-- Wildweasel (vtmldm@epix.net), February 26, 1999.

I have some experience with these embedded 386-based and 496-based boards too. It seems to me that to have real problems, three things must be true:

1) The RTC on the board only handles 2-digit years

2) The software uses the RTC year for something important

3) The software doesn't properly window this year value

I don't have a good feel for how common this combination is, or in what type of application it may be found. I do know that PC BIOSs before 1996 didn't bother windowing the year at all. And most embedded 386 and 486 boards are based on pretty generic PC BIOS firmware.

But I have never seen a noncompliant RTC. Every one I've ever seen behaves exactly according to the specification, and any software written correctly to that specification will sail through the century change without any problems. That correct part is the problem. I kind of doubt that anyone has bothered to write any new RTC-handling code in the firmware for these boards.

As a footnote, I've recently seen quite a few RTC chips in PC boards that have the century in a hardware register, updated by the RTC internal logic. But no BIOS I've ever seen makes any use of this new register for two reasons: the access methods to these RTCs are nonstandard and all very different from one chip to another; and the old method of software windowing must be maintained for backward compatibility in any case.

-- Flint (flintc@mindspring.com), February 26, 1999.


You're right about the key to the RTC issue generally being the RTC handler. If the RTC can be protected by windowing the date info by using a handler patch, more power to those doing it. In fact, eleven years after we found that first Y2K failure, the computer maker involved released one RTC handler patch and second all-new RTC handler to address that problem. In that crash, the RTC's were rendered "un-resettable" and unusable by the 00 rollover, on a five parallel cpu system (ouch!). Five cpu boards later we were up and running to resume testing.

But I do know that Motorola and Dallas Semiconductor have released information pertaining to RTC productss with possible Y2K problems. And in some cases these products have been put on orphan products lists, meaning no factory support for problems. Of concern isn't just those two suppliers, but the second-tier suppliers who produce and market licensed (and maybe unlicensed) copies of those products.

What are these "Y2K hazard" chips from "Joe's Chip Company" being used in? If Asia faces problems with pirate software being a reason many, many computers can't or won't be remediated, what about second- source chips that are used because they're cheaper and more available than compliant versions?

In a world where cost margins drive the manufacturing environment, who's to say these kinds of devices won't show up where they are not expected to? I know that military, FAA and NASA products call for component screening, approval and tracking, so aviation and space systems have such safeguards (we just went through a huge hullaballoo over crystal oscillators being near DOD tolerances, I'd hate to think of something really complicated). Manufacturing experience taught me that auto companies are pushing for component traceability as well, especially in systems that can have liability case potentials (air bags and ABS systems for example).

But I've also seen devices like PLCs which are full of "generic equivalent" chips. And in some cases the chip manufacturers information is blacked-out and the PLC maker's part numbers are applied. Then the question becomes "is this PLC made with parts from ABCD Corp or XYZ Inc and whih version is Y2K compliant?" The PLC maker doesn't know either because their part numbers don't differentiate between ABCD or XYZ. And what applications are such devices being installed in?

To me, embedded devices are becoming more like tip-toeing across a floor covered with hand grenades that have their pins 3/4 the way out. Don't bump one or you'll knock loose a pin, and we all know what that leads to.


-- Wildweasel (vtmldm@epix.net), February 26, 1999.

A more pertinent question might be, what exactly is the hazard in these 'y2k-hazard' chips from Motorola and DS? The worst I can imagine is that they have put century in hardware, and it doesn't work correctly. But this would amaze me -- even the most rudimentary test would show up the problem. I cannot imagine adding a significant feature to the mask without proper simulation, or going to silicon and not validating the package.

I just hate it when a manufacturer says a part is not yt2k-compliant without detailing precisely what this noncompliance is. How can I possibly test it or find a workaround if I don't know what I'm supposed to be looking for?

-- Flint (flintc@mindspring.com), February 26, 1999.


The Motorola info was released last year on their semiconductors devision web site. The list of no longer supported products ran two printed pages. I saw the posting about DS products at work on our department "info board".

As I've thought further about this single board computer issue, one observation came to the fore: problematic SBCs don't employ replaceable proms or flash memory for their bios or operating system, while the applications program is normally on a replaceable or flashable prom. At costs of around three hundred bucks for an SBC, if the bios or OS are problem prone, it's cheaper to replace the unit that to go through the hassle of removing the unit, replacing the prom with one loaded with a good program, bench testing it, reinstalling the device and then testing the system. And if we consider the time the application system is down while an SBC is out for upgrading, it probably is more cost effective to can the unit and install an all-new device.

I've done projects where it was cost effective to replace or reload the proms containing customer applications. These have been PLCs, some 486-based and newer CPU-based SBCs. But whenever it came down to dealing with an SBC that had a known bad or questionable OS or bios, which would include the RTC handlers we discussed earlier, it's come down to "Forget it, replace it." Such software/firmware repair isn't cost effective now in pre-Y2K America. I do wonder about other place around the globe and if it's bad, after Y2K.

When I was in the service, I was shown a miniature (1/2 x 1-1/2 inch) circuit board made in Turkey. It was a replacement for a single in- line package chip that was used as a six-channel, level converter from ECL logic to 28 volts, basically to turn on indicator lights. For the Turks, who were at the end of a long, expensive supply chain for these chips, it was cost-effective to duplicate the workings of the chip with less than a dozen replaceable diodes and transistors.

To us Americans it was cheaper to replace the twenty-five dollar chip. To the Turks, they produced the boards and semiconductors and labor to replace the components as they failed was cheaper. Maybe there's the example here for what we might need to consider for post- Y2K repair projects.

If the manufacuring capacity can't produce new products to replace the failed old products. And the failures won't let the manufacturers run, we're going to find ourselves replacing otherwise non- replaceable chips and upgrading bios and OS code that was abandoned because it does have some Y2K problems.


-- Wildweasel (vtmldm@epix.net), February 27, 1999.

Flint and others,

I managed to dig up the url for the Motorola report. www.mot-sps/y2k/rtcalert.html


-- Wildweasel (vtmldm@epix.net), February 27, 1999.

Wildweasel; I have just read your embedded systems post of 25 Feb. Do you have any insights into back order status of replacement firmware or embedded systems. Will appreciate your response . Thanks,

-- Watchful (seethesea@msn.com), March 09, 1999.

Moderation questions? read the FAQ