What is known of ute embedded systems now vs. early 1998

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

A little over a year ago, I came face to face with my mortality more vividly than ever before. Reading Rick's column entitled "Memo from the Field: Pass the Prozac Please" was one of the triggering pieces of information. The way I perceived the embedded system situation at that point was that our electrical infrastructure was thoroughly riddled with embedded systems of which a low percentage but very high absolute number would fail, in turn causing almost certain failure of huge swaths (or all) of our power grids. My family has been making Y2K preparations for a year and will continue to do so until we've done everything we know to do. But,focusing on embedded systems in electricity, is it now true that the vast majority that "fail": i.e., do not recognize the date properly, will nevertheless remain functional in every important way? In other words, although Y2K may be the most serious world crisis in centuries, could it be true that we are incredibly fortunate relative to how things appeared to be shaping up one year ago? Not that things are sunny, but have probabilities shifted?

-- Anonymous, March 17, 1999

Answers

Bill,

In my experience, what you have stated is true (my experience is testing/remediating Y2K problems in embedded system for an electric utility). We have found no stand alone embedded devices that just freeze up on 01/01/2000. In my discussions with many other utilities, no one else seems to be finding them either. The closest example I have run into is a digital protective relay that will not open when it should (i.e. on a short circuit), but by itself would not directly interupt power to a customer if left unremediated. Obviously, if this description it true it would still need to be remediated, but I have not yet been able to confirm/deny this story.

All of the Y2K failures we have seen on stand alone embedded systems still fall in the glitch or cosmetic failure category. The date is wrong in some way, but the device still functions properly.

Obviously, DCS/SCADA systems fall in a different category. Although still relatively rare, these overall control systems can fail in a manner to disrupt operation of the system they are controlling. The good news is that these fixes are typically just a standard software upgrade that the vendor has already produced.

As you also implied, this does not mean the Y2K problem is not a problem. It just means that one aspect of the problem that could have been very devastating is turing out to be not too bad at all. And no, just because my experience with embedded systems yielded only minor problems doesn't mean anyone else can just ignore their own testing responsibilities.

I hope this helps.

bob

-- Anonymous, March 17, 1999


I would second Bob's response. My experience is as a utility plant engineer assigned to our Y2K project since early 1998. We maintain a database of all assets initially determined to be suspect for possible Y2K problems. In the Plant Embedded category within our database, consisting of 1175 discrete assets, slightly less than 1% (11 assets) have been classified non-compliant after having assessed all assets. When I look at what these are, the non-compliant assets are all PC-based systems or software running on PC-based systems (we use the "Plant Embedded" category loosely). In other words, there is not a single instance of what would be considered an "embedded" device installed in our plant which would fail to function due to Y2K. (In fact, 80% of these assets were determined to be "inert", i.e. not digital or digital but with no date function.) This trend is duplicated in other categories, such as our chemistry lab and with our measurement & test equipment - only PC-based systems and software have been classified non-compliant so far, no true "embedded" devices.

This is not what we expected when we started in early 1998. Having read all the dire warnings, we expected to be changing out more equipment than what we have had to do. These days, the more I read about embedded chip Y2K problems, the more I wonder if it is based on a lack of knowledge about how these devices are used in the real world.

-- Anonymous, March 18, 1999


Bob provided a good answer for the efforts that he's been involved with - but it's important to recognize his closing remark:

And no, just because my experience with embedded systems yielded only minor problems doesn't mean anyone else can just ignore their own testing responsibilities.

It's probably time for everyone to go back and review some history on "embedded controls". This is a very broad category of microprocessor controlled components, and there are so many misconceptions and half truths about Y2K impact on embedded controls (on both sides of the fence).

It is correct to say that not very many problems are being found at a chip level. It is less correct to say, when taking a systemic view (ISA levels 0,1,2,3), that no significant problems are being found in process control environments that encompass all of the previous control system levels.

And it is also true, as Bob implies, that what works for one plant may not for another. There are plenty of DCS's that correctly handle the date transition without crashing the DCS or giving bad data. But there are other DCS's that do crash. We're hearing a lot about a few plants that are operating with an advanced DCS date already. What we're not hearing about are those plants that that rolled the date, and encountered difficulties.

There are many, many DCS vendors. And each DCS is configured specifically for the plant or operations system in which it is installed. They're all unique; there's no cookie cutter answer to the Y2K readiness of a specific flavor of DCS.

So, I think, Bob's answer, taken as a whole, stands as very valid, as does Tom's.

My real concern right now is that such preliminary findings may lead to complacency on the part of those companies who are not as far down the road as Bob's and Tom's are. I wrote this awhile back:

Some companies will get through this because of fortuitious good luck, not because they put a significant amount of effort into Y2k. Some companies will get by through a combination of hard work and fortuitous good luck. Some companies will get hit, even though they've performed very diligent work, because of simple bad luck and problems outside of their direct control. And some companies will get hit hard, because they didn't feel the need to put any effort into the Y2k program.

I'd hate to see what I percieve as a burgeoning sense of complacency, on the part of some companies who were late getting started (or didn't have the executive level buy-in), lead to an increase in failures of the fourth category.

-- Anonymous, March 18, 1999


Thanks, Bob, Tom, and Rick, for reorienting me.

-- Anonymous, March 18, 1999

I am a software technologist and would also like to add a few things. As hardware engineers, you may be surprised which parts of your hardware are actually being used. It is true that sometimes, software technologists rely on "side-effects" of your hardware systems. This may seem like pure foolishness to the hardware engineer, but it isn't to the business. Often, the software technologist's duty is to automate the business process. We may do some "screen-scraping" where we take the date from a display and store it into a database. We do this because the company finds this information necessary for business or automation. This date may then be used to control a security system or an alarm. It may be used to determine when to order supplies. Who knows what it will be used for. You can use your imagination because software is much more flexable than hardware. If you don't believe, ask yourselves why gasoline couldn't be pumped when a single "paging" satellite went out. Without an understanding of the process as a whole, you have very few credentials when it comes to stating how problematic the Y2K problem will be for your company. You must analyze the whole process before you know how bad it will be. As hardware and software engineers, we are merely cogs in the wheels of a system that is controled by management. They must analyze their systems as a whole and then report back to us so that we know how best to prepare. So far they seem to be saying that there is not much problem... If people die because of that, they are in trouble with their companies and their customers, not us. But personally it is a different story. I conclude that since it is unclear to us what their reasoning is, we should prepare our own personal contingencies. After all, our own families are our own responsibility. I would hope that you agree, that your family is more important than your company.

-- Anonymous, March 22, 1999


Now this is very interesting - I read Rick's analysis of y2k in embedded systems based on the current findings, preparing to offer my usual counterpoints, but found nothing to disagree with. Hmmmmm.... at least there's still Drew out there for me to counterpoint! ;)

Regards, FactFinder

-- Anonymous, March 27, 1999


Moderation questions? read the FAQ