I can't stand it. Been reading the posts from today and I just cannot believe the way things are going. Even worse, a lot of it is financial as if we are all believing that there will necessarily BE an economy next year.

The real problem that I see is the embedded chips. Now, before the flamers out there get their typing fingers all twisted up in an agony of anticipation, let me point out that a very small percentage of embedded systems use date functionality. HOWEVER, and it is a big one, a lot of those systems probably *have* date capability, and in my 15 years of embedded work I saw any number of systems which really had no need to be dependent on dates at all, in order to accomplish their primary function, but which wound up with all sorts of bells and whistles which did depend on the date functionality, simply because it was there and could be used.

A Primer on Embedded Systems.

I don't know if anyone has ever set down exactly what embedded systems are or how they work, in general. I haven't seen such a post. So I am going to leap into the breech and provide one.

Embedded systems are those which have a controller chip of one sort or another on a circuit board, inside of device, which control one or more of the functions of the device. Simple enough. They are the chips which run hardware, as opposed to *pure* software, of the mainframe, general pc environment, or the distributed networks, which perform 'calculations', in all of their many and varied incarnations.

There are, to my knowledge, 4 types of embedded systems, classifying them by the nature of the controlling device. Two of these I have experience with. These 4 types are:

1. Microprocessor controllers. These are full instruction set, standard microprocessors, of the same genre as the one in the pc that most of you are using to access this forum.

2. Microcontrollers. These are reduced instruction set, specialized function devices. They are made *specifically* to control devices, hence the name. They have instructions which relate to I/O ports, a feature lacking in the microprocessor, as it has no I/O ports on board. These chips almost always include on-board memory of both the RAM and ROM variety.

3. Programmable Logic Controllers (PLC's). These are a relatively recent addition to the spectrum of controlling chips. I do not know a whole lot about them, as I have never used them. The first mention of them I saw was in about 1993. I am not saying that there weren't any before that, just that they hadn't blipped on my radar screen before then. My impression of them is that they are microcontroller chips carried to an extreme.

4. Programmable Logic Arrays (PLA's). NOT to be confused with PLC's. The PLC is a dedicated control device. The PLA is a Very Large Scale Integration chip. It's primary use was to replace as many as possible of the digital logic chips which were used to 'select' things or actions in circuit boards. I, personally, wouldn't have thought of it, but apparently for EXTREMELY simple applications, they can be substituted for a controller type chip. (My understanding is that, if the control problem can be stated as a complex Boolean function, then it is possible to use these).

I will not address numbers 3 and 4 any further, as they are not something about which I know a great deal. My experience doing logic design as an EE many years ago might allow me to stumble through PLA's, but certainly not PLC's.

However, having done a number of projects in a variety of fields, for a wide array of employers, ranging from the federal government, the DOE, the DOD, medical equipment, telecom, HVAC, and wireless companies, I feel qualified to expound a little bit on the first two. (I know, I know, the only thing I bring to the table is a 'self-reported' knowledge of these things, and 'How come you won't post your real name, and ..... Did you ever hear of a 'Yellow Dog Clause' in contracts? Does the term 'Non-disclosure agreement' ring any bells? And I realize that all I have is 'experience'. Certainly doesn't make me the equal of some of the self-proclaimed 'debunkers'.)

The primary difference between microprocessors and microcontrollers is that the latter were *specificallly designed* to control equipment. The former get 'pressed into service' to control equipment. The former require other chips to actually cause changes of state on hardware, heavy duty relays, I/0 chips, etc. The latter have them built into the chip itself.

One of the distinguishing features, which is directly related to that primary difference, is that, since the microcontroller was specifically designed for the task, all necessary functionality tended to be built into the chip. The ideal goal here was a single chip circuit board. This was a rare bird in the real world, but it did occasionally come to pass. But it was the ideal.

What this means is that, while a microprocessor has its program stored in PROM (Programmable Read Only Memory) or EPROM (Erasable Programmable Read Only Memory) or EEPROM (Electrically Erasable PROM), the microcontroller chip has its program in the on-board ROM (Read Only Memory). Now note the main distinguishing feature here. One of these has multiple types of memory available, all of which have the word 'Programmable' in their names. The other does not have the word programmable in it. The implications here are important. In one case, the chip can be reprogrammed by burning a new PROM and removing the old one and inserting a new one, or by erasing the old one and reprogramming it. In the case of the EEPROM, the hardware may have the correct voltages and amperage to actually do the reprogramming on the board itself and nothing has to be removed! In the OTHER case, reprogramming of the chip involves removing the old one, throwing it away, and starting over. Read that sentence carefully, understanding that the shortest cycle time for this operation that I have ever seen was 18 (That is EIGHTEEN) WEEKS! And that was for a very small scale application. A more normal cycle time for this type of operation is about 12-18 MONTHS, depending on the complexity, size of staff, operating budget, and test equipment availability.

Now, as I understand it, a truly impressive number of organizations are planning on doing 'fix-on-failure'. If the above discussion doesn't make you just a little queasy about this, try thinking what New York City, or Washington, D.C., or any big city will look like after 18 WEEKS without electricity or water, let alone 18 MONTHS.

I can already hear Flint saying 'But what makes you think they haven't replaced those chips?'. Well, I don't know that they haven't. But I do know that there has been no *spike* in demand for embedded designers. I do know that distributed networking applications is paying better than embedded. I do know that 3 former clients have approached me about possibly working on remediation of code, but only in the last two months. And the pay rate was so low as to be ridiculous. Keep in mind, they wouldn't be selling anything new, just making what is already there work after the first of the year. So it is 'maintainance' work, which is the lowest paying. Of course, in one case, replacing the chips would be a neat trick, since the manufacturer left the chip business 6 or so years ago. Which means that the entire system would have to be redesigned, since the chips used had some truly impressive, and non-standard, functionality.

The fore-going discussion, while long, provides much of the background as to why I am literally *terrified* about the upcoming roll-over. These are the things that actually make physical equipment work. Some of it is vital (think power companies, gas companies, water companies, etc), some of it is dangerous (think helium liquefication plants, chemical companies, high speed, high torque fans for air volume adjustment in HVAC), and some of it is just plain necessary (think gas station pumps, medical devices, traffic lights, Command, control, communication and intelligence for the DOD). And I have no evidence that they are being corrected. I have no *evidence* to the contrary, in fairness to the pollies, however, the problem I have is that there are some awfully obvious cues that I should have seen, which would have strongly indicated that something was being done, and these are not in evidence. At the same time, there is some small amount of weaker evidence that nothing is being done.

As the man said, we report, YOU decide. (Fox news).

Me, I gotta go down to Sam's Club, they got a sale on rice and beans.

-- just another (another@engineer.com), October 27, 1999

Answers

LOL @ just another...

Welcome to the crowd my friend. I have tried and tried to tell these pollies that it will be the embeded systems that will do them in, the computers could all be fixed, and the embeds will do the job any way. It seems that some of them cannot connect that the embeded systems control equipment, some of it high speed equipment (which reacts sometimes violently when shut down in mid stride).

And as you , I have looked both fr the obivious signs of the remeadation work being done, and have seen nothing. And I have called several friends to find out what the employers were doing) the utilities companies...Would you believe they are acepting the vendors' word with out making a physical check on their systems.

I have worked for the most part on the coal fird co-generation complexes out in the south west, but there have been a couple of copper smelters, three or four oil refineries etc. That I have also worked on in the last 34 years or so. And yes! I call my self an old shirt tailed electrican. I started out as that and over time went from there.

Untill lately, I have been of the opinion that the embeded systems would crap out over a period of months... Going by my experiences calibrating the PROMS /PLC's etc on the power house / pipline /refineries subsets and their controllers. But the biggest DOOMER of them all, has about convinced me of the TEOTWAWKI senerio. That all the embeds on the millions of other systems globally will go out in just a week or two on either side of the roll over! The concept, because of my experiece, that there would be a prolong period of failures which would allow us a fighting chance to FOF because we would have at least a part of the JIT infrastructure up at any given time. In order to get replacement parts... But an all at once "Turkey Shoot"!

It will be "Total up the bill. Pay off the bartender and turn out the lights" We'll loose it all, and have no communictions to call for parts. No transporatation to get them to us. And no power to put them on line either. I can see us collecively as we all leave the building wistling "Good Night Irene".

~~~~~~~~~~~~~~~~~~~~~~Shakey~~~~~~~~~~~~~~

-- Shakey (in_a_bunker@forty.feet), October 27, 1999.

LOL

Sorry...I forgot to tell you the name of the QUEEN DOOMER !!! Believe it or not? It was Bob Brock who told me that the embeds will all bite the dust in a two week time frame (Man! I sure hope he is wrong) We couldn't come back from a K.O. punch of that kind. Not in time to stop the panic ( figure roughly 6-9 months of FOF work before we could bring back a sembulanc of a JIT supply line).

~~~~~~~~~~~~~~~~~~~~~~~~~Shakey~~~~~~~~~~~~~

-- Shakey (in_a_bunker@forty.feet), October 27, 1999.

Well that about does it.

-- (eat@my.gun), October 27, 1999.

Shakey --

Actually, some of the systems that I have worked on will crap out as soon as the first date that is more than 1999 appears. They have some date arithmetic that will shut down. And thanks to EPA and OSHA mandates, (some of which, admittedly, were justified, particularly safety related concerns), the operators aren't going to make them work even though they KNOW that the reported data is incorrect. The equipment is made to be failsafe and when a sensor reports the wrong thing, or appears to be beyond calibration date, or (fill in any reason why anything might care about a date), then the thing shuts down and will NOT restart until the error condition is corrected. Which, in this case, will be NEVER. Which means you have to replace the stinking chip with one which won't see that particular thing as an error, which translates, a chip with a new program. (See that part about 18 week to 18 month cycle times and picture the Big Apple patiently waiting while ConEd waits for a new chip...)

Others will probably work fine until the first time they are reset or rebooted. At which point one of my favorite little gotcha's will happen. Back then, it was commonplace to lard the initialization of the chip with a whole slew of 'sanity checks' to make sure that the chip hadn't gotten hit by a cosmic ray or something, altering the program. (A legitimate concern, as, for example, the damage that a high speed fan can cause on unregulated startup has to be SEEN to be believed.) And one of those 'sanity checks' involved the date. As in 'date verification', which typically involved making sure it thought it was in the right century, which, of course, meant checking for a 19 on the beginning of the year.....

The difficulty with embedded systems is that they are complex, require comparatively enormous lengths of time to fix, and require hardware. (If the power is out, exactly how does one go about making new chips? take a little tiny chisel, a block of silicon, and about 1000 years....) As for the other types of equipment, the 'true' computers, they too may have problems, which might be remediated after the fact, except that most organizations now run on the extreme ragged edge, (I had a project plan, which incidentally was APPROVED, which called out two key engineers at 36 hour DAYS for 2 months. The deadline was *truly* inflexible, involving as it did a vice presidents bonus check and stock option), and any significant amount of work, over and above what there is right now will probably result in what is known to the trade as the 'snowball effect'. A description would be to take a snowball to the top of a mountain with a very long, very marked downward sloped, snow covered meadow. Place it on the ground, start it rolling, run like heck down the mountain, and try to catch the result. (Hint, don't try this at home kiddies, you WILL get squashed.)

-- just another (another@engineer.com), October 27, 1999.

Well that about says it all. We are dead folks walking.

-- (eat@my.gun), October 27, 1999.

Hey there, eat@my.gun

Why so gloomy guy? After all, a few thousand rolls of tp, couple thousand pounds of rice, beans, peas, some non-hybrid seeds, a way to make sure that the water is drinkable, and a wood-burning stove if you are in the north, a really deep cave if you are in the south, and you ought to easily be able to sit it out until the consumer society makes a comeback in twenty or thiry years.

-- just another (another@engineer.com), October 27, 1999.

Well, you know, I just can't stand beans. More to the point, how can you guys draw such dismal conclusions and then do a LOL? Whistling past the graveyard are ye?

-- (eat@my.gun), October 27, 1999.

Gosh! Just Another...

You sound like old Brock. And of course you know that it means that we are all toast. I well know the lenth of time to re-install the control systems on generation complexes. And it seems that I was right in going on an open ended sabatical last fall.. Joe Six Pack is going to go ape..Well I will be out at the deer lease with my children and their families and a year's supply of food in the RVs. I wonder how many will be left? In about 9 moths from now. Funny thing is...we (start up engineering tried to tell the power that be of the problems way back when we installed and calibrated many of the embeded systems). That there was going to be a problem.."The next generation of controls and main frames will take care of the problem...Don't worry about!" They'll make things right in one of their turn arounds...oopsie!!! Never happened!

Oh well! It has been fun ....

~~~~~~~~~~~~~~~~~~~~Shakey~~~~~~~~~~~~~~~~

-- Shakey (in_a_bunker@forty.feet), October 27, 1999.

-- just another (another@engineer.com), you've done it, again! We've gone totally grey reading your post.

This is exactly why we believe there will be Big Kabooms on 1/1/2000, catastrophic failures. Fireworks to herald the spectacular persistent disasters that feed the 1000s of smaller data-garbled gum-ups.
Devolutionary spiral Spark Starters.

Infomagic, get yourself out of that bug-out bunker long enough to post, zammit. Your time is nigh!

-- Ashton & Leska in Cascadia (allaha@earthlink.net), October 27, 1999.

Aston and Lady...

The situation on the embeded chips has had me frightened to death for over a year now, since I first found out that they had not been fixed. Mr. Infomagic had better get ready for some company, this old shirt tailed electrican is headed out that way, and pulling the dirt road up behind him as he goes. in. Cory Hamasaki has said it correct...Something evil this way comes! I just hope that he goes on to the Barron's place and not wait around at his sites A or B. ~~~~~~~~~~~~~~~~~~~~~~~~Shakey~~~~~~~~~~~~~

-- Shakey (in_a_bunker@forty.feet), October 27, 1999.

Hi "just another",

I'm happy to see your post, as I've been trying to alert people for almost 2 years about the embedded system situation. There's been so much garbage posted by people with limited or no experience with how tiny-scale systems are actually implemented (many times controlled by marketing departments).

BTW, PLC's are fairly easy as long as you know what hardware has to be controlled.

-- Dean -- from (almost) Duh Moines (dtmiller@midiowa.net), October 27, 1999.

Consider the trucks and sprayers that airlines use to apply the stuff that prevents ice from forming on the wings (my technical writing skills are evident here).

Northwest Airlines is on record as saying that without remediation these would not have worked (date function = none).

Now they are fixed (at Northwest) and will work. One less thing to blow up.

-- Me (me@me.me), October 27, 1999.

Thanks for these posts--and please keep them coming. Shakey, I may not always comment on your posts but I always read them with great interest, as I'm sure do many others. Both the official British government site and the well-respected private British site known as Taskforce 2000 are worried about the effects of "renegade" embedded systems. I hope you sprayed some Polly-Off because Cherri's going to be in here like silicon on chips as the self-styled embedded systems problems debunker.

-- Old Git (anon@spamproblems.com), October 27, 1999.

Old Git,

There is only one condolence (sp)here, The posters so far have shown that they know what they are talking about.

Sherri doesn't know a thing about it. She is what you say... self styled kinda like CPR......she thinks she knows. Unfortunately this is the greatest danger all of us face. People who think they know.... kinda like managing companies by the use of buzz words.

I actually feel sorry for her making such a fool out of herself all the time.

-- just*me (justme@justandolyme.meo), October 27, 1999.

"Thanks" for the post, Mr. just another engineer.

Make mine a Mossberg, and fuggeddabout the AMZN puts.

-- um no (&*%^$$$##@^*%$*$ ( .com), October 27, 1999.

Its very easy to claim that certain individuals do not have the experience or training to comment on certain subject areas while not posting proof of your own "expertise". I willing to back up any claims made on this site with proof of education (read MSEE) and job experience - what about you guys or is it just hot air?

-- paul dirac (pdirac@hotmail.com), October 27, 1999.

just another,

As a non-techie, I have no idea whether you're right. But as a writer, I have to commend you for the most understandable presentation I've seen on this issue. Thanks. Please stick around.

-- Thinman (thinman38@hotmail.com), October 27, 1999.

Just another, Shakey & Dean (waving at Dean),

What about all the reports that only 1% to 2% of embedded chips have a date function. Could you please address this issue for me.

Thanks, Cary

-- Cary Mc from Tx (Caretha@compuserve.com), October 27, 1999.

Good post, but, will my VCR work???

-- Uncle Bob (UNCLB0B@Y2KOK.ORG), October 27, 1999.

just another -- it sounds as if you're saying the "cues" you were looking for included ramped-up hiring/subsourcing of designers and your valuable (though) anecdotal experience with some of your former clients, who have mainly punted.

Can you give an opinion from experience as to why the embedded problem has grown increasingly LESS prominent publicly over the past year, after the initial scare. The public line is that enough testing has been done to show that embedded failures will be highly if not vanishingly minimal. Is this a matter of deceit or denial, in your opinion? Wouldn't we expect to see more "fear" leaking out of the profession if that were the case (though it may be happening and be unreported)? Or is this the underlying message the IEEE was trying to give.

Or, based on your experience, is it technically possible that their reported testing results are correct and your own fears are misplaced? And when I say "technically possible", I don't mean theoretically but more at LEAST the 50-50 level of possible?

These are not light questions. I could care less if Cherri posts her dreck or Flint gives us his "maybe-maybe not but Y2K will be insignificant for most people" posts. My remaining preparations could be affected by your answers on this thread. Please stay with the thread until we have hashed this out.

Or, if you would prefer, please email me. I'll keep your identity confidential. Or do both.

-- BigDog (BigDog@duffer.com), October 27, 1999.

I just soiled myself. (While we're on VCRs. Mine keeps blinking 12:00. Is that a y2k problem?)

-- Dave (aaa@aaa.com), October 27, 1999.

just another,

Thanks for posting your experience and thoughts on this complex, and often misunderstood, subject. I'm printing out the thread for closer, off-line study. I hope you and Shakey will keep further discussion public rather than taking it to private e-mail, regardless of any dreck or disruption that may get posted in response. Many of us are capable of discernment and effective noise filtering.

Paul Dirac,

I take it that you disagree with what just another and shakey have posted? If so, please contribute factually to the discussion. By all means, do post your credentials and experience if you think that would illustrate your points and you are comfortable going public. I see no need to insist that others do the same. If anyone is claiming experience they do not have, it should be possible to reveal that through a factual discussion.

Thanks all for a great thread!

-- (RUOK@yesiam.com), October 27, 1999.

Paul Dirac -- I just have to ask, did you read the whole post. There are very good reasons not to publish name, and identifying data. (Those contracts again. That would actually be a good way to shut people whose posts you don't like up. Get them tied up in court cases.) As to credentials, I have a BSEE from a large engineering school. (I don't get into the 'top 10', top whatever game, as my opinion is that once you get past MIT, Caltech, Stanford and U of Chicago, there are about 20 - 50 schools that are equally good in various disciplines.)

Without posting my resume, I will state that I worked mainframes from various aspects including system operations, operations management, system engineering and maintainance, systems analysis and systems programming. As stated in the post, I worked in digital and analog for several years until the increasingly large scale of integration of chip functions in those areas decreased the need for engineers, while simultaneously increasing the need for software engineers, at which point, like so many engineers of my acquaintance, I 'slid' over to the software side, without ever intending to. Since that time, I have worked contract at various places, as I alluded to in the post. Various industries, various product types, both embedded and distributed networks.

Hopefully, this will answer your question. It is as far as I am willing to go though. If the answers don't give you confidence, then take it for what it's worth.

-- just another (another@engineer.com), October 27, 1999.

"Just another"---if what you say is true, then it doesn't matter how close we live to a 711. I hope you are not just some guy who gets off on scaring people.

However, something doesn't compute on this embedded chip issue---if it is truly a disaster, then why are there only a few people on a remote forum that are bringing it up? This should be a well appreciated and discussed issue in technical circles. And it is not convincing to say that that the rest of the world is in denial or that there is a conspiracy of silence.

I hope there are other technically competant folks who will shed more light on an issue that I thought had been shown to be no big deal.

-- Lars (lars@indy.net), October 27, 1999.

Lars, please read this:

Taskforce 2000: Embedded Systems - "The solution is about identifying and managing your risks. This is why the aim of any programme is not to achieve Year 2000 compliance but for your business to be Year 2000 ready."

It *has* been discussed, it *has* been verified, it *has* been covered-over by fear and denial.

Read the conversation between Paula Gordon & Koffinsky:

"'De Nile Aint Just a River in Egypt' ~ Social Pressure, Group Think, and Denial vs Common Sense in the Y2K and Embedded Systems Crisis", Part 6 of a White Paper on Y2K by Paula Gordon, is now posted at http://www.gwu.edu/~y2k/keypeople/gordon

The Truth is out there ... explosive reality.

-- Ashton & Leska in Cascadia (allaha@earthlink.net), October 27, 1999.

---

sobering reads, worth the time

-- A & L (allaha@earthlink.net), October 27, 1999.

You can fool some of the people all the time and all of the people some of the time but not all of the people all of the time.

Just another and Shakeys spiel sounds good, real good, if you arent in the business. But if you are in the business,well it just dont work that way.

Sorry but most power companies are so far behind in their maintenance schedules on a normal basis that if things shut down because they were past their calibration date youd all be sitting in the dark looking at blank screens a long time ago. Way before any of you ever knew of Y2K.

By the way Shakey, if any of your predictions dont come true (and they wont) will you be posting on this board around January 15th or so with OOPS , guess I didnt know what I was talking about statement? Somehow I dont think so.

Can either of you name specific (and therefore verifiable) equipment that has these chips. What we have is this PLC crap over and over again.

Just because the piece of equipment has a date in it doesnt mean the date does anything. Or the time for that matter. What coming up this weekend folks, (besides Halloween). Its the end of Daylight Savings Time. Do you really think we will be sending people out on OT on Sunday morning to change all the clocks so they all agree. And if we dont, why do the lights stay on.

It aint the software Children, its the engineering.

-- The Engineer (The Engineer@tech.com), October 27, 1999.

Folks, The Engineer is a shill and a troll, and does work with the power system in our neck of the woods. He's got real narrow tunnel vision blinders on. Part of a PR campaign. Did you enjoy Richardson's (sp?) visit?

If we are wrong about Y2K outcome, we'll meet you for Crow. But that is not likely to happen. Please stop your Doubting Thomas aspersions.

-- Ashton & Leska in Cascadia (allaha@earthlink.net), October 27, 1999.

Shill?

According to my dictionaries definition a shill is a confederate of a gambler or carnival barker or peddler who pretends to buy something in order to lure onlookers into participating. Im not selling any preps, gold, generators, etc so I think that is a completely incorrect use of the word.

Troll?

One definition of troll is to sing lustily in a full throated voice. Ok, Ill buy that.

Blinders on? Au contraire. I was worried about Y2K years ago, before, Im willing to bet, you or many people on this board ever heard of it. But the more I looked into it, did the research (and not just surfing the web) and some testing the less and less I become concerned. I have a BE In EE and 26 yrs. of experience, and thats hands on experience, behind me. You have? Ive put the stuff in and made it work. SCADA, microprocessor based relays, breakers, etc. And you know what you know????????????? Please enlighten me.

I didnt go (to Richardsons (SP) visit). I actually do work here, not just PR.

Oh by the way, Portland General released a statement saying that each year they have about 6 power outages over NY eve due to drunk drivers hitting power poles. They expect more this year because of more people out driving. If the power goes out in your area will you automatically think its Y2K? Would you believe anything else?

Meeting me for Crow? I dont think so. I think you will justify anything that happens as being Y2K related or a cover up. As Flint has said all sides will claim they were correct.

-- The Engineer (The Engineer@tech.com), October 27, 1999.

See folks!?! WE TOLD YOU a month ago they were going to blame it on drunk drivers!

BBWWAAHHAHAHAHAHAHAHAHAHAHAHAHA

It hurts to be right.

-- drunk terrorist squirrel woodchuck watch (allaha@earthlink.net), October 27, 1999.

Just Another Engineer,

Was not questioning your education or experience (within that post) - I directed that message to the individuals who were premptively "slamming" another person (read Cheri). I do not know their target nor her background or if it qualifies her as a embedded's expert. But than again lets hear what their qualifications are and some proof to back it up. Myself, I have worked within the fields of integrated circuit design (Cypress Semiconductor, Inc. and Motorolla) and had stints employed for EPRI as a Researcher and Hughes (digital control theory - PLCs). Educational Career - MSEE (OSU)and BSEE (TAMU). Regards,

-- paul dirac (pdirac@hotmail.com), October 27, 1999.

The International Union of Operating Engineers has information about Y2K on their website at http://www.iuoeiettc.org/y2k/

Also, IUOE has invited the public to a free two-day Y2K seminar in Beckley, West Virginia tomorrow (28th) and Friday (29th). Here's your chance to ask the engineers what they think about Y2K! For more information about this two-day Y2K conference entitled "Y2K Workshop: Final Preparations to Protect Workers, Families, and Communities" contact the Y2K Program Coordinator at 304-253-8674. (Or post to this thread & I will answer questions if I know the answer).

It will be held in the auditorium of the National Mine Health and Safety Academy, 1301 Airport Rd, Beaver, W.WA starting at 8:30 AM. Speakers include Dr. Paul Hill, chair of the US Chemical Safety & Hazard Investigation Board and Dr. Fred Millar from the Center for Y2K & Society.

-- d (d@d.com), October 27, 1999.

From the IUOE website (above):

"Isnt the Y2K problem greatly exaggerated?

No one knows how serious the problems will be. To be safe, stationary engineers should share the experts concern about the interconnectedness of systems in their facilities. Schindler elevator control systems, for instance, do not rely on dates to control operations so they should pose no problem. However, if those elevators are connected to fire and smoke detectors, security systems, timers or emergency power systems, they can be shut down by Y2K problems in the other systems.

The government and private companies have spent billions of dollars getting ready but it is clear that smaller firms and organizations are not as ready as larger organizations. The Building Owners and Managers Association (BOMA) conducted a survey of 937 of their members and reported in mid-year 1999 that 25 percent did not have a formal written plan for addressing the Y2K problem. Only 47% reported completing their assessment phase, and 23 % reported completing their remediation phase. These may be facilities that stationary engineers work in every day."

-- d (d@d.com), October 27, 1999.

Just Another,

Question: My local utility here in Raleigh (CP&L) has stated that all of their production facilities have been "rolled over" for a number of weeks now and that they are *already* operating in a virtual post-rollover environment. Could you comment on this? Is such a thing possible? Is it significant? Does it eliminate some or all of the uncertainty at actual rollover? Any analysis appreciated.

-- Puddintame (achillesg@hotmail.com), October 27, 1999.

All -- Will try to answer each as I can...

eat@my.gun -- Don't recall saying 'LOL'. Actually, I believe I said I was *terrified* of the situation. Don't really think it is a laughing matter.

Shakey -- Yeah, I seem to recall a few of those 'Don't worry, the (fill in the name of the hardware, software, or system) will be long replaced by the time *that* could ever be a problem.

Ashton & Leska, Glad you liked the post. Sorry about the impromptu hair color job. Purely unintentional.

Dean -- That is part of the problem. Too many people, including some who should know and choose to ignore the issue, don't seem to understand. And one thing I know is that it is hard to make decisions with inadequate information.

Me -- Good for Northwest. Now if *everybody* with potential problems does same, we'll be in good shape. BTW, they didn't happen to mention what it was that would have killed these systems in the absence of dates, did they?

Old Git -- No Polly-off. Opinions are like armpits, most folks have a couple and they generally smell to anyone else. I just ignore it. Glad you liked the post.

um-no -- Sure hope it doesn't come to that.

Thinman -- THANK YOU!!!! You got the point of the post. All I was trying to do here was to try to make this rather technical area a little more accessible to the layman. Apparently, from your reply, I got close. It is the issue that must be understood, as some apparently missed. Again, thanks for seeing that. Makes having spent the time to do it worthwhile.

Cary -- Okay. I would personally have put the number a little higher, more like 3-5%. I don't know the methodology used to come up with the number. I would venture to guess that estimates were done on functionality. The pitfall here is one I attempted to address in the original post. The term for it in the industry is 'feeping creaturism', a spoonerism for 'creeping featurism'. The symptoms are additional features to a device that has no need for them, which are added simply because they can be (e. g. I had a calculator once that if you pressed a certain combination of keys simultaneously, came up with an 'evaluation mode', which included a little 'dot race' game. Useful? No. Why was it in there? Because they could.). I have seen this happen over and over. Usually driven by marketing. A couple of the projects I worked on had about as much need to use the date functionality capability that was available on the chip as a submarine needs a screen door on the pressure hull, but, because it was available, we wound up larding the thing with stuff to flash the date on entry screens, output screens, anywhere and everywhere. Shoot, the device could be used as a clock! Useful? NO. Why? Because we could. A pure 'functionality' analysis probably wouldn't catch these sorts of things which would lower the tally a bit.

OTOH, I seem to recall seeing the results of an empirical analysis carried out in some county down south, Georgia, maybe?, where they went around counting the systems with embeddeds, and then tried to make a determination of which ones would fail. Seem to recall they came up with some number between 3 and 6%, and a sort of 'random' distribution, i.e., it didn't actually make much difference who the manufacturer was, or what the application was.

Another point to ponder is a 'system' can contain an awful lot of chips. I have worked on some that contained upwards of 10,000 separate controllers. And the problem with the really big ones is, that any one of them can sometimes shut the whole system down. (This is a real factor in places where safety is an issue.) In which case, it doesn't matter that the other 9,999 chips are compliant, if the one that isn't takes the system down, the others might as well be non-compliant too, at least until the thing is back up. This is part of the 'avalanche' or snowball effect. Fix this chip, bring up the system, down it goes, go find the next one that's broke, fix that, repeat until the end of time....

Hope this answers the question, if not, clarify and I'll take another swing.

Uncle Bob -- It depends: Fault tree analysis - Does it work now? N - Diagnosis: Probably operator malfunction. Read the manual. Y - Good, Do you have a generator? N - It will work if and only if the power is on. Y - Good How much fuel do you have for the generator? Some - When the fuel runs out, so do the taped Gilligan's Island showings. LOTS - Good

Big Dog -- Okay, this may take a while and we may go back and forth a bit. First, I don't think that the former clients necessarily 'punted'. What I do think is that they waited till awfully late in the day to start looking at the problem. And this may be symptomatic. Consider. Of the embedded systems I have worked on all were of type 1 or 2 in the original post. Most were of type two. These chips are 'fire and forget'. You do EXTENSIVE testing with bit boxes *precisely* duplicating the operation of the equipment, under every scenario you can think of, then you make a prototype sized order of the 'finished product' chip, with the program in the ROM in order to go to full scale prototype testing. Then and only then do you order the production run. The program is simply a binary bit pattern created at the same time the chip is made. A little hard to 'fix'. The system, whatever it is, gets installed, initialized, tuned, and *that's it*!! It is now expected to run 'forever', with possible tuning adjustments. This means that some of this stuff may well have been 'forgotten'. (BTW, I do know of at least one case where this happened. The system works well and there was no reason to look at it, shoot, they hadn't even tuned it in three years. Just dumb luck that somebody picked up on the question of 'Wonder if it is Y2K ready?')

This may explain some of the 'less prominent' aspect. Another point is that this tends to be a pretty tightly knit clan of folks doing this. There aren't all that many who do it. There are even fewer who really like it. The mind set doing this is *completely* different than that of folks doing *pure* software. (Not better, not worse, but very different. Before all the mainframe, pc, and distributed networking folks cloud up and rain all over me.) It would frequently take me a two or more weeks to 'shift gears' when transitioning from an embedded job back to a *pure* software job. And most of these folks are at the *bottom* of the pecking order. They can't see the overall picture any clearer than I can, or Ed Yourdon, or you, or Shakey, or the Engineer. We see our own little pond and if there aren't any ripples in it, then there aren't any northern pike in the world. The next one over may have a hundred, but we don't know that. And even if ours has problems, we tend not to think that anybody else has problems. (This is an aspect of engineering that people do not know about, generally, many of us tend to be less than confident in our abilities, at least privately, regardless of the face we put to the world. I remember once, one of the other engineers got all over me for being to deferential to the other folks in the dept. "You've got just over 3 times the programming experience in C of the entire rest of the company combined, and you're gonna let them make stupid decisions because you're 'outvoted????!!!" This kind of thing is relatively common. And in my experience, the ones that come across with the 'I am ALWAYS right' attitudes are the ones you have to watch closest, because they tend to be the source of the biggest errors. Overconfidence is a terrible thing. Thus, a lot of engineers will take the position that "Well, *I* may be having problems, but those geniuses who are working for 'XYZ' just have to be a lot smarter than me and the guys I work with, and surely they aren't having any problems. This is a point of psychology, true, but valid, in general, nonetheless.)

As for the *official* line about enough testing being done and so on, I cannot tell you. As I said, I've been working in the distributed networking applications area the last 5 years, with only a tenuous connection to embedded. (The company's products include embedded chips, but I have been working stuff like build management, makefiles, software and document configuration management and release management, the 'behind the scenes, nitty-gritty detail work of making sure that we can actually ship what we were supposed to.)

Generally, it would depend on the type of chip. Again, I cannot speak to the type 3 and 4 chips. Never worked on them, and although I suspect that they probably are not that far from the types I worked on, I don't wish to speculate where I have no expertise. One of the replies I got to this thread, from Dean, suggests that he may understand these and be able to give us a better glimpse of the 'man behind the curtain'. But for the ones I do know about, the type 1 and 2, it depends on the type of chip in the system. A microprocessor can be tested fairly easily. Because these typically get their programs from another chip, the PROM, EPROM, or EEPROM, one of these can be burned with a replica of the original code, the micro can be convinced that the date has rolled over, and you go from there. At that point, it is simply a matter of exercising each leg of the code and seeing what it does. (Like the whole problem, it isn't the complexity of the task. It is the sheer volume, the repetitive nature, and the lack of people with the ability to do the task.)

The other type, the microcontroller is a different story. The jobs I have done involving these type chips, the testing was a VERY involved process, done long before the first product hit the street. It was MUCH less expensive to buy a single chip computer, with all of the memory, I/O control ports, etc, on it, and the program burned into the ROM, than to put together a mixed mode system using a microprocessor. And once you had the product out, you were probably not going to be doing anything to it until the next major upgrade, which in many of these cases would be 5 or more years out. So the testing hardware, etc, got subsumed into other projects, frequently with modifications. So the test equipment may not be there. The chip itself usually isn't used for this, instead and emulator for the chip is used. These systems are expensive, and take programming themselves, in many cases. Thus, I have trouble seeing just exactly what test methodology might be used to check them. (BTW, if there are any embedded test folks out there who can give us insight into this, I, for one, would love to hear from you!) I am not saying that it can't be done. Just that I am not sure what they would do. And at the very least, I would expect such a process to be fairly labor intensive, and drawn-out.

As for why there isn't any press about all of this, well, it may be the psychological point I brought up earlier. It could be that nobody believes they have a problem. It could be the standard argument: What is the downside of making this problem public? Ans. -- Our stock will tank NOW instead of in two months. We, the executives on whose watch this occurred are likely to be fired. Our stock options take a beating. Our bonuses take a beating. The companies credit dries up and we have an even harder time addressing the issue. The company, executives, and insurance company may be involved in lawsuits. Now, what is the upside? ........... Anybody? ........... Come on, speak up! ........... Meeting adjourned.

This is my best guess on the publicity issue.

As for the testing results, of course they could be right. 50-50? I don't know. Don't know what the methodology used, don't know what sort of management is involved, whether it is third party, or what. I don't have enough information. That having been said, I repeat what I said about cues in the original post. Maybe there have been a whole lot more engineers come out to work on this stuff. That *could* account for some of it. (This is at odds with the industry wide complaints of lack of engineering talent.) But again, this is anecdotal evidence, although I will point out, that the ones I know of and mentioned, I know of myself, NOT second or third hand. It just seems that not enough time was available.

I'll go out on a further limb here. All of the reports I have seen say that there has only been about 1/3 of the money originally budgeted for Y2K spent. Now, in all of my years as an engineer, I personally have NEVER been on an *over-budgeted* project. I do not know anyone, personally, who has ever been on an *over-budgeted* project. I have never *heard* of anyone on an *over-budgeted* project, and don't know anyone who has ever heard of one. FWIW, and again, it is an inference, not direct evidence, but what I am being asked to believe here is that Y2K, which looks like some large number of software projects, comprises the first set of projects in all of history which were over-budgeted. (And over-staffed, etc.) All of the however many hundreds of thousands of projects all decided, independently, that this time it was different, this time they would get all the money, time, staff, resources, .... that were needed. Somehow, this does not inspire confidence in me. But again, it is speculation. (I can categorically state that it isn't every project, the ones I know of where I work are not in this category. Then again, they aren't done either, which may color the viewpoint.)

Having said all that, here is the rub. Microcontroller projects lead microprocessor projects approximately 8 to 3 in my experience, with 4 mixed type projects. That is, on the jobs I've been on, 8 of them were microcontroller based. 3 of them were microprocessor based. 4 had both types. So out of 15 jobs, 8, or just over half, were purely microcontrollers. This may mean that my experience is 'biased' toward this type. And to further confuse things, one of the 'mixed' types had over 10,000 processors and controllers, as well as both digital and analog hardwired controls. So beware of quoting this as a 'definitive' statement. This is *one* persons experience. It does NOT necessarily reflect the state of the rest of the universe. (How's that for a disclaimer, Flint?)

I'll be happy to respond further if I overlooked something, or if something requires clarification.

One last point. Like almost everybody who actually understands the *potential* of the problem, I DO NOT KNOW WHAT IS GOING TO HAPPEN!!! I don't have a crystal ball. Those who think there is something definitive in this, there ISN'T. There can't be. I don't believe that there is ANY single person in the WORLD who has the complete overall picture. Everyone, and I do mean everyone, who touches this mess finds themselves in the position of the three blind men trying to describe the elephant. (Sorry for the diatribe here, but it needs to be said. All I am trying to do is to point out that here is what looks to me like a great, big, gaping hole in the bow of the ship that appears to have occurred when we struck the iceberg, and do you think it might be a problem, captain?

I am exhausted from answering the Big Dog, so will cover any I have missed later. (The wife is calling me for dinner, and I don't wanta miss those survival beans and rice, which she has tricked up to look, taste, and feel, EXACTLY like a steak, baked potato, and salad!, MMMMMMHHH!)

-- just another (another@engineer.com), October 27, 1999.

Puddintame,

The issue of CP&L advancing the plant operating dates should remove those systems which have been advanced from the pool of potential failure points on rollover date. Not to say that some of those systems wouldn't be affected or disabled by a failure of a non- advanced subsystem.

My brother in-law who is a plant operator for CP&L confirmed an earlier story about the plants being advanced (as far as 2003 in some cases), But he stated that "they still don't know what's going to happen".

This thread is dead-on, Embeddeds ARE the Y2K wildcard because they are a complete unknown.

WW

-- Wildweasel (vtmldm@epix.net), October 27, 1999.

Lars --

Actually, the intent here is not to scare anybody. The intent is to get information to those who may reasonably use it, if they are so inclined. And as to the 'bleak picture', yes it is, but that doesn't mean we should all give up saying 'It doesn't matter how close we live to a 7/11". (In point of fact, I live 38.7 miles from the nearest 7-11. I measured the distance the first time I heard about Milne's trademark signature.) The POINT is that if what I fear is correct, and, let me emphasize, all I can do is point out the possibility, and why I think it might be a problem, and why if *I* think it might be a problem, you might want to consider it one too, but if it is correct, what it *does* mean is that the '2 week' paradigm is a load of crap and the thinking has to stretch a little longer. Where I live, growing season begins about late March, early April. Meaning that we have to be prepared to get by until the garden starts bringing in produce in late May or early June. We need a longer term solution to the problem of water than a couple of gallon jugs in the basement. We need to find a way to handle sewage and trash disposal over a longer term. These are NOT problems without solutions! The solutions may be unpleasant. As an example, if this scenario rings true to you, you just may want to rethink those 'neighborhood sharing' scenarios that I have seen. Run the numbers (for example, my neighborhood, in sight of corn, pumpkin, cabbage fields, has approx. 108 children below the age of 16 on the street facing, the street I am on, and the street behind, abutting my backyard; how many hundred pounds of rice and beans does it take to feed them all until spring?) Calculations like this become very sobering, very quickly, and if ANY of you think I *LIKE* the conclusion, YOU ARE INSANE!! And that is putting it mildly. But it is also a real fast reality check. But those who will not face these sorts of questions may be in for a real bad experience in the near future. Scary? Dambetcha! Is it my intent? No. Go on back to sleep if you wish. But consider that if you have not even *thought* about the possibility, if all of your time and effort is put into questions about how do I handle the 0-10 scale economy-based scenarios, then you have essentially abdicated your chances of handling the situation gracefully if it is otherwise. Only you can decide whether or not to give any credence to the issues I have raised. And believe me, I do understand that from your point of view, I could be anyone, literally, including some creep who was making it all up to scare people. Honest Answer? God, how I wish I was.

As for the other issues. On the question of why isn't it a public issue. Let me refer you to the answer I gave Big Dog. There are any number of reasons why it might be. I do not know if any, or all of them are close to the mark. My personal best guess is that is a combination of the entire range of possibilities. Many companies may truly not believe they have a problem. An old favorite expression of mine is that 'Nothing is impossible. (for the man who does not have to do the work.)' I personally am doing a 'command performance' the evening of rollover, and when the question of exactly what it was that they thought I could do if the power goes out, or if the servers quit, or the vendor tools turn out to be not quite as compliant as they have assured us, or if it is our add-ons, if the compilers, assemblers, linkers, and loaders turn out not to be quite such holy writ as we have been led to believe, the answer was (and I quote) "Oh, you'll think of something."

In other instances, it may be a case of 'Okay, we have serious problems, but then again, we're probably all dorks, those guys over at (name of other company here) are all geniuses, they'll figure it out, and all of their stuff will work. It'll just take us longer.'

In still other cases, it may be that the management meeting to discuss early disclosure went something like the one I described.

Furthermore, you might look at some of the links provided by Ashton and Leska, or 'd', or look at the IEEE site for their discussion. These provide what you may wish to think of as more 'open and unbiased' discussion.

The biggest problem that I see is that there probably isn't anyone, anywhere with the perspective to see the overall picture. And that is a largish problem, in and of itself. (I went into this at some length in my reply to Big Dog. I am sure that some of the 'flamers' will put this down to 'CYA'. I really don't give a rip. The entire point is that I look at the government's point man and he is exactly what *I* would have picked as *MY* point man on a technical subject.... A LAWYER. Some of the Y2K policy at the company I work for is driven by.... (yep you guessed it) A LAWYER. Other portions are driven by marketing. My guess would be that folks in these positions would be the ones to see the overall picture. But they most often don't have the technical expertise to understand it. And they almost inevitable DO have the expertise to understand the FINANCIAL ramifications of any potential decision to 'go public' if there is a problem.

I hope that I have allayed your fears that I am simply some 'scaremonger' trying to drum up fears among the gullible for some dark, nefarious reason known only to himself. (Which I suspect could ONLY be known to himself. After all, we aren't sitting around a campfire telling ghost stories. There isn't a lot of personal interaction, you don't get to see peoples faces turn white or whatever, so I am at somewhat of a loss to fathom the motives of anyone who would do this sort of thing. I mean, isn't the reality bad enough? )

If not, then take the post for what it is worth.

-- just another (another@engineer.com), October 27, 1999.

I'll go along with this. But exhaustive testing of every device is probably not cost-effective for most companies, because the cost of the testing will exceed the cost of the failures you prevent. Type testing is very device dependent. Where implementations can be variable, it's not good enough. Where they're basically all alike in operation, just testing one or two is sufficient (assuming, of course, that the type designation includes the firmware version, such that two versions are considered two different types).

Of course, beyond the issue of what constitutes sufficient testing is the question of how much testing has been done, or results of testing elsewhere of the same device acted upon if necessary. Even if someone spent the resources to fine-tooth an essentially similar facility elsewhere and made their results available, someone *still* needs to go through your own facility carefully, checking off each device. Inventory and assessment, in other words, are still expensive and time consuming even if someone else has already done most of the work.

And just how much of this has actually been done? My guess is, not enough. Things will stop or go boom, here and there.

-- Flint (flintc@mindspring.com), October 27, 1999.

The Engineer --

Okay, so you are an operating engineer at a power company. And your company is all ok. Glad to hear it. I don't believe that I stated categorically that all power companies are toast. I believe I stated that I DO NOT KNOW. And that most of them HAVE embedded chips. I didn't work on any significant FRACTION of the embedded systems in the world. Sheesh! I'm one guy. I've worked on about 20 - 30 projects in my career, depending on how you define them, whether you count call-backs to clients for upgrade work a new project, or whether new feature development counts, etc. And only about 15 of them were direct embedded design.

However, I will point out that my end of this was DESIGN. Not operations. Not testing, although I have done a fair amount, particularly pre-production testing, as outlined in the original post. And not remediation work. How much clearer do you want it. No, it surely doesn't work that way everywhere. Why on earth would you believe that it should? Although I would be willing to bet that no matter HOW far behind yall are on your maintainance, the SAFETY stuff gets recalibrated on time. Or have you somehow bribed the OSHA inspectors. And the same goes for the EPA stuff.

And just how much testing did yall do, hmmm? Test every leg of the code? All of the exception handling routines? All of the reporting routines? Or did you just roll the main system clock forward and nothing happened and the bills go out on time and therefore it is all A-OK? (That last, BTW, appears to be how the local power company did it. Really gives me a nice warm feeling. This same bunch of morons can't keep the juice flowing reliably when everything else is WORKING. And they are worried about their billing system. One of their generation guys, during casual conversation mentioned that they had checked everything and it was ok. And when the subject of steam valves came up the reply was, 'Oh, why would we check them? They're *automatic*.' Nice to know, and I kindly refrained from asking if that meant that there were elves in there to adjust them. NO, NO, I will not ask where you live and what power company you work for. NO NO.)

Sorry to be nasty like that, but in my humble opinion, you deserve it. The point of this thread, at least from my point of view, was to define the problem, and point out some potential pitfalls in the solutions. Apparently, from your point of view, it was to provide you a forum to show off your abyssmal ignorance.

-- just another (another@engineer.com), October 27, 1999.

Puddintame --

WildWeasel has answered better than I can to this question.

WW --

Thanks for the assist. (And good info). I do not have personal knowledge of ANY power company. (Other than the local one, as I mentioned in my response to the Engineer, and that is based on casual conversation, and their normal ineptitude.) I have knowledge of about 15 organizations of various types. Some I know for a fact have a problem. Others I suspect do. A few I have no idea on earth where they are on this. The point I was trying to make is that THIS is how the stupid chips work. This is what has to happen to remedy the problem. As far as I can tell, it doesn't appear to be happening.

I tried to word it so that some poor person who hasn't got a PHD in engineering, with a hundred years of experience in the field might have a chance to understand it. So that they could make up their OWN minds about whether it is deserving of consideration.

-- just another (another@engineer.com), October 27, 1999.

Flint --

Good to see you weigh in. As per usual, you have some valid points to contribute.

The problem is that while it may not be cost-effective, it is the only way. The reason being those cycle times I mentioned. THOSE are what I perceive as the real problem. If it wasn't for that, FOF would be adequate. But if you haven't fine-toothed your facility, or you are relying solely on letters of compliance from the manufacturer (and did you get one from the company that made the chip? the company that wrote the program? Do they still have the source code? How about the test records?) then if you find a failure which causes shut down, then you are shut down for a LOOOONG time! And thinking about where-all these things are, when considering 18 week downtimes, much less 18 months, the mind boggles!

Nice response. Thanks.

-- just another (another@engineer.com), October 27, 1999.

just:

Yes, the actual failure mode is critical. Some things you can live with as is, some you can hotwire around, some you could have fixed quickly but the collateral damage means rebuilding the facility! Most of the time, but not all, you can derive the worst-case failure mode from the function of the device itself. As you move up the embedded "chain of command" the failure modes become worse and worse, since they are more global. It's not the sensor in the chimney you worry about, or even the controller returning bad data from that sensor, it's the server deciding what to *do* with the bad data that you have to watch out for most carefully. Again, we'll miss some. And we'll hear about it.

-- Flint (flintc@mindspring.com), October 27, 1999.

The Engineer (<---love that handle. Sure like yourself, huh?)

Hi, I am The English Teacher. And I am here to give you a little lesson. When Ashton and Leska called you a troll and you said it is defined as "singing in a lusty voice" or whatever, you missed the point. "Troll", as they used it, is a noun. You mistakenly defined it as a VERB.

So you might want to check with Mr. Webster again. Look for the definition that has the little (n) before it, okay? Great, thanks a lot.

just another engineer--I am JUST an English teacher, but when I first learned of embedded systems a year and a half ago I was forced to get it. Someone told me the satellites up in our atmosphere have embedded systems....is this true?

-- preparing (preparing@home.com), October 27, 1999.

Flint --

Yep, the failure modes are important. Now, I don't know what you have seen out there. The ones I worked on, virtually ANY time there was a safety issue, or an environmental issue, the answer was 'fail safe'. That is, any exception which even remotely touched on these issues caused a 'graceful shutdown', except on initialization in which case it would complain about the specific failure and then refuse to start. (Beats heck out of the alternatives.)

And you are right. Some things you can live with. If an MRI doesn't work, okay, well you don't get your picture taken today. Some things you can hot-wire around. If your inventory robot won't go get the stupid part, then take a basket, walk down the aisle til you get to the proper bin, grab one and bring it back. And some things are a problem (and boy are you right, will we ever hear about it!). These are the ones that concern me. I don't believe that, right now, anyone has a particularly good handle on which are which, or what the consequences are. Which could lead to resources being squandered on the air-conditioning systems, while leaving the railroads to flounder , etc.

I will have to disagree with your analysis about the food chain hierarchy though. The situation you describe is only one of the failure modes. That 'sensor' chip down at the bottom can occasionally shut the whole thing down. Take as an example an AC system that is on a microcontroller. Limited memory, so they load, say, the enthalpy table in segments and 'bracket' or 'window' around the current date to determine which segment to load for a given day. (Mind you this is all very theoretical and completely hypothetical. Don't know that the example is real, in other words.) So when rollover happens, and the date arithmetic screws up, since it doesn't know what a 'negative' date signifies, it loads up what it thinks is the right segment, and it is for summer, and the time is winter, so it goes way the heck off to the right on the table looking for a valid number for calculation purposes, and doesn't find one that makes sense, and so reports, given the sensor reading that it has, and the entire unit shuts down, because it thinks that it is below absolute zero, and KNOWS that can't be right, and it's instructions are 'if it even verges on stupid, shut down until some human being can fix it', which they can't because all the readings are already correct, and the only way to 'fix it' to the computers satisfaction would be to hold a blow torch to the temp sensor, or hold the humidity sensor under water or something...

So it kind of depends on what the system is doing, and whether or not you run into what someone thought was a boundary condition, or a fatal error, etc. The big problem is that there is no way to KNOW short of exhaustive testing of each and every chip in each and every system, including every version that is in the field, just how much of this sort of thing may be waiting to happen.

Other than that, I would agree with your analysis. Just pointing out that the 'hierarchical' view of systems is only valid to a point.

-- just another (another@enginer.com), October 27, 1999.

preparing --

Not necessarily any such thing as 'Just' a teacher. A good one can make kid. A bad one can break one. (I was fortunate that two of the few teachers I can remember taught English. And two of the best profs I had in college taught English Lit, World Lit and Shakespeare. Having time to do so, I took a bunch of English courses, because I liked them. {Never could understand why all the other engineers thought I was weird..}) Off topic, I know, but I thought I might share that with you.

The answer to your question is an unqualified yes. They are, by definition, embedded. (At least, I don't THINK there is anybody up there with a keyboard and monitor running them. ;-))

These things are all over the place. Satellites, undersea telephone cables, deep sea oil rigs, pipelines, you name it. They are such USEFUL little buggers. They are PARTICULARLY useful in situations where the equipment to be controlled is at extreme remote locations, or places (like space or the bottom of the ocean) where people are at a bit of a disadvantage. Silicon, unlike human flesh, does not care about pressure, at least not to the same extent. Doesn't care about atmosphere (or lack thereof). Or cold, or heat, and ... you get the point.

-- just another (another@engineer.com), October 27, 1999.

This is an excellent thread in an awful kind of way, one of the best in the past few months. Forgive me for spoiling the lovefest just a wee bit but my family is at stake, minimally.

Last I heard from Flint in any definitive fashion, he stated that Y2K, in his opinion (fair enough) would probably not be noticed AT ALL by most people. Yes, I know he's not "sure", but that's what he said.

Now, keep in mind Flint is a hardware engineer. I have never thought he understood software at all but have listened to him pretty carefully about embeddeds.

Flint's BITR confidence (remember: he is right down the line with Hoffmeister) is a far cry from "just another" who reports himself as *terrified*. Sadly, I have read nothing so far ON THIS THREAD in the way of rebuttal by Flint or anyone that would suggest his *terror* is misplaced. Nor have I read anything to suggest his expertise is being substantively challenged.

Yes, I know that "just another" ALSO doesn't know what WILL happen. In fact, I'm at least as chilled by the evident care with which he clearly treats his own experience (and the limits of that experience) as I am by his specific concerns.

He began this thread as follows:

"I can't stand it. Been reading the posts from today and I just cannot believe the way things are going. Even worse, a lot of it is financial as if we are all believing that there will necessarily BE an economy next year."

I have earned my stripes as a doomer, yet I smirked inwardly a bit when I read that sentence -- "sure". Then I began considering his comments.

Bottom line? I take full personal responsibility for all preparation. I'm about to take on some additional responsibilities.

Can someone tell me why this is *misplaced* terror?

-- BigDog (BigDog@duffer.com), October 27, 1999.

Big Dog --

Thanks for the reply. And if you get something that explains why it is misplaced PLEASE let me know too? (That is the main reason I go on any of these forums. I've been looking for that very thing desperately for over a year.)

By the way, thanks for understanding the point of the post. Same comment as to Thinman.

-- just another (another@engineer.com), October 28, 1999.

Just another--

I agree with various others that this has been a stellar thread, so much so that it has made me break out of the "lurker" phase I've been in for the past six weeks or so. I don't have any technical background in this field (Ph.D. in English from the U. of Illinois), though my concerns about Y2K and the power industry (among other matters) have led me to correspond occasionally over the past 18 months with such folks as Rick Cowles, Mark Frautschi, Phil Hystad (the main designer of three SCADA and EMS systems used by U.S. power companies), etc.; and I've tried to understand a few of the basics involving DCS, SCADA, EMS, and assorted Y2K issues at the generating, transmission, and distribution levels. (Also very valuable was the ABB report authored earlier this year by lead scientist Dr. Klaus Ragaller, and a discussion I had one pleasant evening about it on the old euy2k.com forum with Malcolm Taylor, a New Zealand power company engineer.) On broader Y2K issues I've corresponded on a number of occasions with both Mr. Yourdon and Dr. Yardeni.

Generally, I've been "middle of the road" on Y2K re the U.S.; I did hit an 8 on the ole WDCY2K scale earlier this year, but more recently have been in the 6-7 range (cf. Bruce Webster, co-chair of the WDCY2K Users Group). But there has been quite a bit of disturbing news coming down the pipeline in recent weeks, as even Flint would attest. Also, I keep thinking back to a statement issued on June 30th by Mr. Cameron Daley, chief operating officer of TAVA/R. W. Beck (which has since been merged into the Belgian giant REAL software group), a statement that was carried by "Bloomberg News" and that was posted in the "news" section of Cowles's site. (In fact, I'm the one that confirmed the authenticity of the news article for him by checking the "Bloomberg" archives.) And here's where I have a request for you, Just Another, with your obvious technical expertise in embedded system design.

Mr. Daley, a former exec with Boston Edison (I have no idea what his technical background is), stated that, based on TAVA's work with over 100 U.S. electric utilities, he considered that *regional* power outages up to several weeks in duration were possible. In his view, various distractions such as deregulation were causing some power companies not to check their systems widely and deeply enough. Drew Parkhill of CBN News later did a brief telephone interview with Mr. Daley and reported it on Cowles's forum: Daley stood by his remarks in the "Bloomberg News" article and stated that when such power companies said they were ready for Y2K, they weren't lying--they were just wrong. In other words, it was a case of ignorance, not deception. According to Mr. Daley (Parkhill said he offered a few "technical examples"), they were missing problems that could have directly led to significant outages.

Now, at the time Mr. Cowles and I thought that this was simply another case of a critic pointing out that some power companies were perhaps relying too much upon type testing (against which even NERC has warned) and vendor compliance statements (consider that telecom Bell South found up to a 50% error rate in the statements of its vendors!). But, getting edgy, in recent weeks I've been wondering exactly what Mr. Daley had in mind. Parkhill (a nontechie journalist) was supposed to try to do a follow-up interview with Daley, but to the best of my knowledge he never did. I emailed Cowles (who used to work at TAVA before starting his own company) about the possibility of his contacting Daley for a technical discussion, but he never replied to my request; perhaps his parting from TAVA was not entirely amicable or perhaps he has concluded that Mr. Daley was off base in his remarks. I'd considered contacting Daley myself, but it would be much better if somebody with technical expertise did so. I don't know his email address, but it probably could be obtained by contacting TAVA: info@tavatech.com

If Mr. Daley's remarks were firmly rooted in discoveries made by TAVA's own engineers, technicians, and systems analysts, that's troubling news indeed. TAVA is a leading firm in IT and automated system research, consulting, and remediation; TAVA is the primary consultant to GM, Weyerhauser, Kraft, Chevron, the USPS, and assorted other big-time outfits. Their product/embedded system database now runs well over 110,000 items. (You might want to do a little exploring at their website: www.tavatech.com/home.htm) In August 1998 I looked at their online assessment data for four giant U.S. corporations: an auto mfgr. (GM), an oil company (probably Chevron), a pharmaceutical company (probably Johnson & Johnson), and a beverage company. TAVA was finding 15-20% failure rates in plant floor embedded systems--presumably mostly LSES, or Large Scale Embedded Systems; GartnerGroup, the IEE, the British Office of Health & Safety, etc. have all cited failure rates of up to 35% or more in such systems (without remediation), systems which are heavily involved in process, control, and safety functions, especially in the petrochemical industry, etc.; without remediation, odds of total plant shutdown were 60-90% in each case, according to TAVA.

So I'd sure like to know what the good folks at TAVA have found (or think they have found) with regard to certain power companies, if the info can be generalized enough to get around confidentiality restrictions. I get the distinct feeling that TAVA and EPRI should be comparing notes but probably aren't. If you feel like trying to contact Mr. Daley, please do so and report back to the forum. Otherwise, I may try, though because of my nontechnical background I'm really not qualified to do this in a sufficiently detailed manner.

-- Don Florence (dflorence@zianet.com), October 28, 1999.

There are some very interesting arguments in this thread, and some even more interesting conclusions. I cannot dispute anything that "Just another" has said for the simple reason that I have not worked on the same systems that he has, nor have I ever seen any code that he has written, or had any reason to test the results of failure on any of the embedded systems that he talks about. For the same reasons, it is doubtful that anyone else can argue against, or support anything that "Just another" has commented on.

However, as someone who has over 25 years experience in power generation and distribution, and is actively working on Y2K issues within the power industry, I can comment on effects of embedded systems failure.

Within our own group of power stations, it is a common misconception among the IT staff that a failure of an embedded sytstem due to Y2K issues will cause the plant to shut down. In fact we have found almost no items that would cause a shut down on failure. The sole exception was in an old GT station that is due to be de-commisioned, and I have already disscussed this system in detail on the elctricity Y2K forum. We have tested our generation system by rolling the date forward on the network, on the SCADA systems, and on any PLCs which have any date/time functions. On any systems that we could not determine if there was a Y2K issue, we shut the system down to see if we could manage without it. When our SCADA proved to be non-compliant we ordered and installed a new SCADA system which brought its own problems. We ran one of stations on manual for 19 days and found that there were no difficulties in doing so.

While I am convinced that we can manage our own power stations without any major problems during the roll-over, I cannot and will not try to assure anyone that ALL power stations throughout the world are equally as ready. However, I have not heard of many major issues with oter plant. Perhaps "Just another" could enlighten us with some more details of the plant that he believes might fail, and why. It is possible that he does have information that we have missed.

Malcolm

-- Malcolm Taylor (taylorm@es.co.nz), October 28, 1999.

P.S. Correction: Going back and checking the text of the original June 30th "Bloomberg News" article entitled "Utilities Say They're Y2K Ready, Though Blackouts Expected," I find that Mr. Daley did not actually use the word "regional" in his remarks. Here are the relevant sections of that article:

"'Utilities are scared,' said Cameron Daley, chief operating officer of Framingham, Massachusetts-based Tava/R.W. Beck, which tested and upgraded systems for more than 100 U.S. utilities. 'The whole grid won't collapse, but there will be outages that could last up to several weeks.'"

"Still, deregulation has pushed utilities to cut labor and other costs, and those that are deepest in negotiations with regulators haven't been as focused on preventing problems related to the millennium bug, Daley said."

"'The utilities most distracted by deregulation aren't doing enough to identify and prevent problems,' Daley said. 'There are a number of instances where utilities didn't go deep enough into their systems- -they accepted vendors' words that parts of a system were compliant.' Even if a utility corrects all the problems in its own systems, power still may be cut off to their customers. That's because U.S. and Canadian power lines connect all utilities, and when one utility system breaks down, it could cause problems for others."

The article itself had been pasted on an old Cowles forum thread, where I also found Parkhill's posts about his telephone interview with Daley. Daley confirmed the accuracy of the "Bloomberg" quotations, said Parkhill, who added: "His basic point was that utilities *believe* they are being accurate when they report readiness, but his company has been called in to audit many of them, and has subsequently found that they (at least some) have not always gone deep enough into their systems, and that they have missed Y2K problems that actually do exist--problems which could indeed lead to power outages." In another post, Mr. Parkhill elaborated on Daley's comments in the phone interview: "According to my notes, he said that on 'numerous occasions' they had been asked to audit power companies, and that in 'a lot of cases' they found that the company hadn't done their remediation well enough. The basic problem was the companies hadn't gone deeply enough into their systems. Those undiscovered problems could lead to outages."

Now, either Daley and TAVA are right, or they are not. I do wish some of the resident engineers on this forum would try to get at the truth of this matter.

By the way, in support of one of Mr. Taylor's points in the post immediately above, I recall a post on Cowles's forum some months ago noting that a Pacific Northwest hydroelectric plant was able to operate manually without SCADA for 18 days without problems. My layman's "read" on this issue, from the ABB report and elsewhere, is that serious SCADA and EMS problems *could* lead to some grid instability through operator errors (manual operation through communication back-up systems), given that some variance in frequencies and loads could arise, but that outages in such cases are not inevitable. One of my main questions regarding Daley's remarks is whether he was concerned about undetected Y2K problems in SCADA and EMS, and in DCS in certain power plants. Because of all the excess generating capacity in the U.S. in January, I'm personally much less concerned about scattered problems in individual generating plants than possible problems in T&D systems.

-- Don Florence (dflorence@zianet.com), October 28, 1999.

Any thread that draws out Don Florence and Malcolm Taylor is definitely unusual. Where is Rick?

While hoping "just another" will take a shot, speculatively (given what he has already caveat-ed) at Malcolm's question, I'm prepared to take Malcolm's general postings over the past year seriously. He has always struck me as one of the very few pollies who CARES about everyone's profound concerns and who is dead-honest in worrying about things HE may have missed. And whether he is a polly IN GENERALY about Y2K has never been addressed by him, I believe.

As another layman in this area, I remain somewhat confused by claims that the grid is as complicated as a moon shot and yet somehow so simple that (to listen to some) Y2K remediation on all levels has been trivial -- yes, I oversimplify, but the contrast has been painted frequently. Something doesn't jive.

Anyway, the reason I digress is that, vital though utilities are, they are scarcely the entire picture with embeddeds.

Oil. Natural gas. Chemicals. Wastewater. Nuclear. Telecomm. Emergency response systems. Medical. Manufacturing. .....

So far as I can see, we are still not only exposed to significant individual failures but to one or more sector-wide failures. "Exposed" does-not-equal "prediction", "exposed" equals reasonable "terror", given the implications of such a failure.

Why?

Because FOF on three days, already ludicruous for software (a thread is coming, but I've promised Lane Core to wait until he publishes something on this), is not only logicaly but physically impossible for embeddeds (think supply chain as well as the fix-test-break-fix cycle that "just another" outlines). A broad sector-wide failure (oil? chemical? pipelines? ports?), requiring months (?) to fix, will see wondrous human ingenuity -- and terrible human suffering.

Malcolm --- could you address the disconnect between grid complexity and simplicity a bit with respect to failure? If you are willing to comment speculatively about embeddeds in other sectors, that would be very interesting inferential material for some of us.

Without embeddeds, I would have prepped entirely differently. As WW and so many have said over the past two years, embeddeds are the wild card. They dwarf Hoff's function points or, on the other side, Yourdon's sofware deja vu.

I quite understand that no one can predict embedded impact calculatively and that we may well differ about the risk of these exposures. I began my preps with the thought that a 2% chance of TEOTWAWKI was already WAY too much. Silly though the numbers are, my current 5% "terror" is way WAY too much. And, worse, a retrospective will show that my terror should have been rated at, say, .01. Or, perhaps 50%.

Hey, "just another", wait up, I'll meet you at Sam's Club.

-- BigDog (BigDog@duffer.com), October 28, 1999.

Awesome thread, folks. I'll just throw in another two cents worth.

Was talking with a real engineer from our local power company yesterday (I'm trying to set up an interactive hookup with them for my PV system), so, of course, we eventually got around to the subject of 'will the lights go out.' His answer: "I don't know. We think we will be OK in this valley, where we have an old analog power plant, right next to its own coal mine, but we have no idea about anybody else, or the rest of the grid. There are too many unknowns: Deregulation, mom and pop IPPs, Wheelers, embeddeds, etc.; any one of them could potentially take down the grid, or parts of it." I said, "Can you isolate?" He said, "Maybe."

Then he said, " I'm more worried about oil."

He did not have even one scintilla of happy face BS for me.

I went home and put my nose back to the prep grindstone.

FWIW

Godspeed,

-- Pinkrock (aphotonboy@aol.com), October 28, 1999.

I'm "just a local pastor". I've been doing Y2K "briefings" in churches for the past year and a half.

Before a local meeting I sat down with our power company IT people and did a 1 1/2 hour interview. They were very straightforward. (They also happened to be personal friends.) Here were their conclusions:

Solid thumbs up: Customer billing was ready to go.

Solid thumbs up: Admin 'stuff' was ready to go.

When I asked if we would have power, they weren't so positive. All they could muster was, "We sure hope so."

I asked about embedded systems.

"Uh...we don't really know. We'll have to wait and see like everyone else."

So....if you don't really know, how bad could things get?

"We could be out for maybe up to two weeks, max. But, we can always go manual."

The conversation went on for about another 45 minutes. As we were leaving, I asked what their personal assessment was...

"24 hours, max. No big deal."

To put it bluntly, I left the meeting wondering what these guys had been smoking. And these are good friends; intelligent, sharp, "know what they're doing". I got the feeling that this thing is so far over their heads they don't know what knowledge base to draw from (if they have one to draw from). In other words, it sounded like it went beyond the scope of their expertise.

They found their level of incompetence. I was not comforted.

I've asked a lot of folks, from all over the world, in diverse businesses about embedded systems. I have not heard one, concise, specific estimate. Obviously there are too many variables.

And that is why I am preparing.

God's best to all.

-- Chris (chris@lifetel.com), October 28, 1999.

just another,

"I tried to word it so that some poor person who hasn't got a PHD in engineering, with a hundred years of experience in the field might have a chance to understand it. So that they could make up their OWN minds about whether it is deserving of consideration."

Thank you for taking so much time to answer mine and other laymen's questions. I may not have a PHD in engineering, but I'm certainly capable of separating the wheat from the chaff. I come to this forum searching for quality posts with real information to help me complete this Y2k jigsaw puzzle. To the others who have shared their knowledge on embedded systems on this thread, thank you also.

-- Cary Mc from Tx (Caretha@compuserve.com), October 28, 1999.

I have a possible "dog that didn't bark" datapoint to toss out for consideration, relating to embedded systems and the electric utility industry. Might be good news.

At the beginning of the year, the potential for a "common mode failure" seemed to be very high on NERC's list of things to worry about. I took it to mean, for instance, a commonly used piece of equipment that would misbehave systemwide either because it was not known to have a problem or because not enough of them could be found to replace. I understood the associated contingency plan to involve minimizing the need for energy transfers across the grid.

I have not been able to find any reference to common mode failure in NERC's most recent documents.

Makes me wonder whether NERC has truly discarded the possibility of a common mode failure (based on work since the beginning of the year allegedly showing embeddeds will not take down the grid).

Or is NERC just not openly referring to it as a reason behind the continuing recommendation to minimize energy transfers.

Or perhaps I have not found the right document.

NERC's website can be found at: http://www.nerc.com/y2k/

-- Brooks (brooksbie@hotmail.com), October 28, 1999.

I'll second BigDog's assessment of Mr. Taylor's posts. They are uniformly excellent--and very readable, which those of us without technical training appreciate greatly.

Mr. Daley *might* have looked at Y2K problems in power companies undergoing TAVA audits and concluded a bit too easily that those problems *could* or *would* lead to outages. Again, I don't know what Mr. Daley's specific technical background is, or what the lead engineers, technicians, and systems analysts at TAVA are saying. That's why it would be nice to have this issue fully aired (or aired as much as possible, given confidentiality clauses) and discussed by knowledgeable engineers, etc., in an open forum. I will add that, in Y2K discussions, it is all too common for "could" to become "would" and "might" to become "will"; there's a cautionary tale in that for all of us, as NBC in three weeks will hit the American public with a "Y2K" movie virtually certain to be loaded with melodramatic exaggerations. The ABB report made a point of repeatedly stressing the robustness and resiliency of power grid systems--a very important point to keep in mind.

At the same time, it's really difficult to quantify exactly what risks do remain. For instance, the CIA, NIC, and State Dept. all still seem to think that some overseas power systems are at serious risk of failure from Y2K problems, which suggests that Y2K problems can, at least theoretically, take down some power systems. I find it difficult to believe that Lawrence Gershwin, science and technology officer for the CIA and NIC, hasn't ever gotten on the horn to EPRI, TAVA, etc., to find out just what the technical risks are considered to be. Mr. Gershwin is surely basing his assessments on some reasonably reliable technical knowledge.

In his T-100 Y2K internet audio conference, Dr. Yardeni pointedly asked a State Dept. official (as I recall, Inspector General Jacqueline Williams-Bridgers) whether the State Dept. considered that Y2K no longer posed a significant risk to various power systems overseas or whether this was still a "gray area." "It's still a gray area" was the reply. That suggests that there are still some unresolved or uncertain issues involving technical Y2K risks to some power generation and delivery systems--and if some power companies in the U.S. have not been as thorough in their remediation and testing as they should have been, then I suspect those issues and concerns would also apply to them. Again, one would like to know, in understandable detail, precisely what problems TAVA has been unearthing in its power company audits.

No doubt we'll all have this issue resolved for us one way or another in just a couple of months anyway. But it would still be worthwhile to have as much reliable information ahead of time as possible.

"Knowledge is power," Bacon observed. Ah, it would be nice if knowledge were also electric power!

-- Don Florence (dflorence@zianet.com), October 28, 1999.

The embedded problem appears real, though we can argue about failure rates. My foremost concern is fuel. How small a failure rate will take out oil wellheads and refineries? BTW, the US has blown up ten to twelve of these this year already, presumably trying to fix them. My best guess after 1/1, virtual shutdown of world trade, virtual cutoff of new fuel supplies, collapse of Russia's power grid and subsequent meltdowns of their nuclear plants, implosion at IRS and USPS, and, in short order, a global earthquake as the vast debt/credit balloon deflates at mindnumbing speed in 'cascading crossdefaults'. If this isn't enough for Teotwawki, what is? I submit a theological thought: replacing men's mucles with machines was one thing, but replacing men's minds with artificial thinking machines quite another; God may be angry, a bit, at humankind's arrogance--do we really think we can imitate his most remarkable creation? He's not going to send down thunderbolts, he's just going to watch as our hubris blows up in our faces. And, no, I am not 'conventionally' religious.

-- stanley heidrich (heidrich@presys.com), October 28, 1999.

Just another.

Operating Engineer? Tisk Tisk. No Im a real engineer. And operating engineer (also called just operators in some companies) is someone whos job it is to run the system on a daily basis. Think blue collar like an electrician. Im white collar. No, not being a snob just telling what the differences are.

Test every line of code? No why should we? The thing you dont get is that the time and date dont have much, if anything to do with the actual running of the system. Its determined by load and the response to that load. We dont know what the specific load will be at a specific time. Its variable. You can use statistics to come up with patterns and averages but not specific numbers. If you could get everyone in your neighborhood to turn on everything they own and then turn everything off the load would change and the system would respond to it. It wouldnt matter if you did this at 5:00 PM or 5:00 AM.

The problem is you dont really know why dates and times arent important. Except for billing. But billing isnt running the system. So you go into all of this stuff about PLCs with chips, blah, blah, blah. The date doesnt have an effect on the way the system operates. It was never set up that way. Timing is important. Time isnt.

Oh and by the way Ive worked in design, construction, and maintenance. When you talk about testing every line of code and exception routines and reporting routines its obvious that you dont know how the power system is set to function. You are assuming its like the industry you know. Wrong.

Maybe your operating engineer buddy meant they were mechanically automatic. Lots of stuff is still electromechanical. No chips in them at all. Lots of things are 30 or 40 years old out there.

You werent defining the problem. You were defining your definition of the problem. Not the same thing kid.

Oh and by the way they do care about temperatures, etc. Look at the rating of all the equipment comes with. I pulled a book down from my shelf for a piece of equipment we use and its ratings are 40C to 70C. Wide but not infinite. Most places that have a lot of this equipment usually have heat pumps to control the ambient temperatures. There have been cases of equipment enclosed in modular buildings that stopped working in the middle of summer if the AC failed. Interesting that you dont seem to be aware of that.

Sorry if you dont like my attitude but like Malcolm said IT people tend to think that embedded systems are the end all and be all of everything. The electrical grid is NOT the same as a computer network.

Preparing (or should I call you The English Teacher?) Get a sense of humor will you. Lay it in with your preps. And as for your learning about embedded systems a year ago and now getting it. Well..lets just be glad you are teaching English and not Science.

Brook: The reason common mode failure isnt very high on NERCs list is because there is so much different equipment out there used in different way. For instance most power companies east of the Mississippi use Power Line Carrier and Blocking schemas on their relays. Companies west of the Mississippi tend to use whats called Permissive Transfer Trip. They essentially have the same function: To clear a line in the case of a fault but the logic used is different. Also there are lots of different SCADA vendors out there and its used in different ways by different companies. Malcolm said that his SCADA system was non compliant so they replaced it. Ours was fixed but wasnt replaced. Also while it was technically non- compliant the problem was such that if we turned it off and rebooted after the roll over (takes about 3 minutes) the system would have worked just fine. We opted to fix it anyway. So what may affect X may not be a problem with Y. And then you have to add in the fact that the type of scheme used to protect a distribution network isnt usually the same thats used to protect the High Voltage bulk transfer network.

-- The Engineer (The Engineer@tech.com), October 28, 1999.

Engineer, you are a total ASSHOLE. You are a LIAR too.

-- Pissed Off (pissedoff@rage.com), October 28, 1999.

I'd feel better about the "Engineer" if h/she didn't claim to be omnipotent about everything.

Don -- You've put your finger on the question I have for Malcolm and couldn't phrase as well. Puddintame has often called attention to it as well. The general line here in the states seems to be that utilities were NEVER a problem BECAUSE of their very nature (see the "Engineer" for the PR). Yet, apparently reputable sources see utilities overseas as seriously at risk.

Huh?

Yes, I know one can always plug in the "so would we be if not for remediation." But that's not the spin I hear. And if this be so, then we're back to wondering about the near-totally self-reported nature of the remediation. As Rick Cowles, who is moderately optimistic about the grid, says, the utility industry is second to none in its overall arrogance.

And I'll repeat again that utilities are only one embedded sector. The chemical sector may well turn out to be the real Y2K trojan horse.

-- BigDog (BigDog@duffer.com), October 28, 1999.

Engineer - Thanks for your response, but it doesn't quite do it for me.

NERC's earlier reports indicate that common mode failure WAS very high on their list of concerns but the more recent reports indicate it has either fallen off (for good reason) or been kicked off (for less admirable reasons) NERC's radar screen (or that I am not finding the right documents to compare). My question is why, since it might indicate good news.

(Your comment about the difference in technology between east v. west of the Mississippi supports the notion of gridwide susceptibility to common mode failure, just not so likely nationwide.)

-- Brooks (brooksbie@hotmail.com), October 28, 1999.

PO: Your opinion of me doesnt matter. However if you could point out a specific where I am lying please do so. If I am lying (and Im not) I must be lying about something. So what is it that Im am lying about?

Big Dog: No, you mean omniscient not omnipotent. Oops doing it again, arent I? And no, Im not claiming either. But I certainly do know more about how the power system works (and doesnt) then just about every poster on this board, possibly excepting Malcolm. What ticks ME off is how the posters seem to know things about a business that they never worked in, dont know much about and yet believe all the negative stuff and none of the positive.

Especially when their information seems to be second or third hand at best and are always in the form of: I was talking to my friends brothers who has a second cousin twice removed who has a neighbor who has a friend who once worked for a power company somewhere or other and he says.. Jeez give me a break will you.

While I have posted (rarely) on other topics Im as entitled to my opinion as anyone else. Especially when most of you seem to be posting with great assurance on things you are absolutely clueless about. Technology isnt faith.

No, no one said that utilities were entirely not at risk. Things had to certainly be checked out. I know from talking to an engineer at a very large coal plant in the Southwest that they did find things that would have caused trouble. They did this quite a while ago by the way. The misconception is that there were a lot (verses very few), that utilities ignored the problem, and that they are covering up. As Ive said previously: The utility business if very conservative. We didnt (despite some writings to the contrary) automate everything back in the 50s. There is still a ton of old stuff out there that doesnt have any chips in it at all. I know of utilities that have relays older then 50+ years still in service.

Brooks: No its not a difference in technology. The relays tend to be the same. Its the difference in how the technology is applied. Common mode failure is a worry but not as much as what are called second and third contingencies. If you go to the WSCCs web site youll see that when you look at what caused the big outages of a few years ago that was it. Its if X goes wrong and then Y goes wrong, etc. Thats really the hard thing to figure out.

Common mode is like saying you have a relay made by the ABC relay company and all of a sudden you find out that each one has a fatal flaw so all of your relays wont work. The way around that is to have the primary set from ABC company and your back up set from the DEF company. Or have model X as set 1 and model Y as set 2. As I said there are a lot of different vendors out there and not every company everywhere has the same type and vintage of equipment and they dont all apply it the same way either. So the idea that there will be one crucial thing that everyone has overlooked it a big stretch.

As an example SF6 gas which is in a lot of breakers has a low freezing point. Even lower when compressed. Most places dont need to use heaters in the tanks. But if you are a power company in Montana or Maine it is something that you have to think about. Florida or Texas, no. A heater problem could cause breakers to not operate in the former areas but not the later. So the idea that if there was some heater problem every utility in the US (or the world) would be trying to get the problem fixed at once isnt valid.

-- The Engineer (The Engineer@tech.com), October 28, 1999.

"just another,et al" A great post and one that has been badly needed. Although it is a long one, I hope some of the intellegent questions posed by many participants will be answered in short order.

Hey "Pissed Off", I can understand your frustration with some on this link but name calling doesn't accomplish a whole lot, really nothing. Let's try to elevate our conversation.

My feeling is that when we discuss a particular issue such as embedded chips or systems, as good as the discussion is, it is important to apply it to other areas of industry as some have indicated. "The engineer", for example. You may have solved all your utility plant problems where you work but what if telecommunication don't work? Won't you be up the creek without a paddle? Do they not have embedded chips in their systems? What assurance do you have that they will be able to supply services to your utility?

We need to get the blinders off and begin to connect the dots. Utilities need telecommunications and telecom's need electricity. It looks like a catch 22 to me. And doesn't this complicate the issue of electric services provided to us all? This embedded chip issue is too complex and pervasive, in my opinion, to be adequately solved by 1/1/2000.

BTW, "just another's" description of embedded systems was excellent and, as a follow-up, I would recommend Bruce Beach's description regarding Secondary Clocks as a "must read" as well. It can be found at: http://www.webpal.org/Gas.htm.

Meanwhile, I am going to continue to prepare for what portends to be a "dicey" year 2000.

-- Jim (jwworden@efortress.com), October 28, 1999.

there are only 64 days to go. if it ain't fixed, it probably wont be. we all want to be convinced that there will not be a problem, but there is enough misinformation and misrepresentation coming out that we really can't believe anything. if you have doubts, and if your on this site you do, prepare and pray. that's all we can do.

-- robert paul brock (rbrock401k@aol.com), October 28, 1999.

A final note. Some grid systems in various undeveloped and developing countries are rather unstable even under "normal" circumstances. Mr. Yourdon, who has worked on five or six continents, has described the experience of working in a country where one does not know from day to day how reliable the electric service will be. For such systems, a given set of Y2K problems might easily trigger outages. On the other hand, the far more stable grid systems of countries like the U.S., NZ, etc., might weather the same level of Y2K problems quite successfully, all other factors being equal. To use the language of the ABB report, the grid systems of some countries are obviously much more "robust and resilient" than are those of other countries. This may explain at least to some extent the discrepancy between "foreign" and "domestic" assessments of Y2K risks to grid systems.

Anyway, for any given grid system, the main question seems to be this: At what point would an accumulation of individually small problems (undue frequency or load variances, or whatever, from whatever causes) become sufficiently severe to trigger an outage? As I recall, the ABB report tended to indicate that it would probably require an unfortunate combination of factors in addition to Y2K itself to trigger a major outage in a "robust" grid system. But, as I noted before, it seems difficult to quantify such risks. For that reason, I'd still like to see a discussion by knowledgeable engineers, etc., in an open forum of whatever problems TAVA is finding in its power company audits.

-- Don Florence (dflorence@zianet.com), October 28, 1999.

Jim,

No. We have our own communications network. Mostly Microwave along with radio and fiber. Most other utilities have their own (the big ones anyway). A lot of it is PLC (Power Line Carrier). There is also a back up in that if the communications are lost equipment such as relays go into back up modes. Then they work by time delay (as in timing not absolute time) rather then awaiting a signal. All the plants will be manned incase the AGC (automatic generation control) which does need communications is off line. Plus it would all have to go out. Telephone, radio, cell phones, internal phone system, external phone system, etc. You also have to add in the fact that the phone networks are anticipating problems due to the large volume of traffic with people calling each other. So wed have to allow for that alone if nothing else.

Plus most of the Telecoms have said that their phone system (the important part anyway) also doesnt depend on dates and times.

By the way Beachs article was totally wrong. Not just a little off. Way way way off. His secondary clock thing is from the Twilight Zone. And that is one of the problems: Stuff like what he wrote sounds plausible but in reality its just hogwash.

-- The Engineer (The Engineer@tech.com), October 28, 1999.

The Engineer is correct with his last statement on telecomms not being reliant on embeddeds (that utilize date stamp function). Most Wireline and Wireless Providers will be affected primarily in Billing and Operations System Support (OSS)...The networks can be managed without OSS funtioning as earlier network management functionality still exists that is not date dependent. Yeah, most Pwr Companies employ microwave links (read Teligent, Spectrawave, etc) to supply their point-to-point connections - probably formatted as a T1 (or E1 if in Europe or Asia) to handle data/voice communications. Microwave links are fairly robust as they employ +1 redundancy (i.e., spare radio at each tower) or employ a ring topology that can compensate for a site going down.

-- joe thomas (jthomas@excite.com), October 28, 1999.

what I would REALLY like to hear from someone in the utility industry is "I've tested EVERY chip, even though the vendor stated that there was not going to be a problem" also "We rolled the clocks forward on every mainframe And EVERY chip at the same time." we usually only hear "we rolled the clocks forward", never explaining "which" clocks. ALL clocks? can you even do that with chips?

-- bob brock (rbrock401k@aol.com), October 28, 1999.

"The Engineer" Thanks for your quick response. It is easy to write someone off with non-specific statements and obviously, there are those who do that to you. However, why is Beach's premise "totally wrong" and as you say "Hogwash"? How about some specifics? Anyone else want to comment? Also, how does your utility plan to protect itself from the potential cascading effect that could be produced by a grossly non-compliant utility that goes down on the grid you are attached to?

-- Jim (jwworden@efortress.com), October 28, 1999.

Hey The Engineer....English teacher here. You said it is a good thing I am teaching English and not Science since it took me a year and then some to get it after learning about embedded systems.

You scare me. You are an ENGINEER on this forum trying to debunk other engineers and you can't even read a post carefully enough to understand it????? I said I GOT IT as soon as I learned about embedded systems over a year ago. Meaning I GOT IT right away.

By the way, how long have you known about embedded systems? When do you think you will be GETTING IT, my friend?

Hee hee. Couldn't resist.

-- preparing (preparing@home.com), October 28, 1999.

Jim,

Just a quick suggestion about Beach: do a search in the archives under BEACH. You will see quite a few threads with a LOT of discussion about his "theories".

I am a non-techie, but I found that his critics made far more sense than Beach did. But, find out for yourself.

To everybody who keeps banging away at the Engineer: why can't you cite somebody who works in the electricity industry to refute the Engineer's claims? I'm sorry, but I can't buy the claims of those who do not have direct experience in that industry when the subject is so technical.

-- Johnny Canuck (j_canuck@hotmail.com), October 28, 1999.

The Engineer knows sooo much about the electrical grid and now he is a know-it-all about embedded clocks. He has blown his credibility with me.

I re-read Beach's report and it is not hogwash .

The big engineer's opinions did not make me feel relieved about this embedded systems potential problem one bit.

Trust the engineer ? I don't think so.

-- An Old Electronics Man (Been@AroundaLongTime.com), October 28, 1999.

let me see if I have this right. "just another engineer" (all lower case letters, very humble), actually has programmed chips. "The Engineer" (I thought there were more than one) has never programmed them, but has tested them. Has he crunched any code? did he do any of his companies' remediation? Mr. "The Engineer", what is your qualifications in telling us what is going on "inside" the little black boxes. It's what you DON'T KNOW that we worry about.

-- bob brock (rbrock401k@aol.com), October 28, 1999.

And when you realize that ALL of the above is AMPLIFIED by the woefully inadequate Priority Interrupt Structure of Intel's Chipset...

It's going to be Yumbo next year!

-- K. Stevens (kstevens@ It's ALL going away in January.com), October 28, 1999.

All --

Just got home from work and found all of these follow-up posts. I'll try to get to all of them. May take a while.

Don Florence --

Sorry, I am going to have to decline to follow up. For one thing, I know very little about the power generation industry. Or the grid. Waaaay back when I was in college, I had power, fields, computers and controls as options and chose the latter two, then had to drop controls for want of time. And I haven't kept up with the other two since. (Had all I could do to stay current in my own field.) I suspect that there has to be someone better qualified than I to do the follow-up. For another thing, I don't have the time. Been as busy as a one legged man in a butt kicking contest lately, and it doesn't project to get a whole lot better between now and year's end.

The points and questions you raise are good ones. Some of the numbers you quote *really* surprise me. Although I suppose that they shouldn't, since they are quoted for 'large scale embedded systems' which I would assume probably means multi-controller systems, and the more controllers the greater the opportunity for a problem. You might see the discussion between Flint and I about this issue above.

Sorry I cannot take up the gauntlet.

-- just another (another@engineer.com), October 28, 1999.

Malcolm Taylor --

Thanks for the post. A ray of sunshine. I sure do wish that you *could* state categorically that other plants would. I'd sleep a little better.

As for some plant I know of, as I pointed out to Don, I don't know anything about the power industry. (I suspect that this area got jumped on because it is a real obvious one where one 'awshit' cancels out a whole lot of 'attaboys'. The sector is a key. If it survives then, there may be a fighting chance to correct everything else, while if it goes, then we probably don't have a prayer of doing anything constructive about other problems. Probably couldn't even see the other problems. Unless someone has a propane powered computer out there?)

The point of the thread was about embedded systems in general, and was an attempt to frame the problem from the perspective of an real-time embedded designer. The power industry is one of those that use embedded chip technology. I suspect that to some extent, folks may have taken the words 'embedded', 'fix-on-failure', and 'problem' and got the bit between their teeth and took off with it, as it is an 'outfront' target. My concern was more of a general problem with embedded technology as a whole. The cycle times to repair these are a lot more than the three-day weekend being bandied about. And these types of chips are (or were, if what I have read here is correct, maybe they all became PLC's, or PLA's while I was off doing distributed network stuff), ubiquitous. They control a lot of things and represent at least some of the productivity gains we have seen in the last decade.

If the question at the end about a specific plant is from the back and forth with 'The Engineer', this is not a 'factual' thing. I didn't go there and look. It was casual conversation. Although the general level of ineptitude by this particular company makes me wish I had a good generator and about 100 years of fuel for it. And that is if Y2K is a negative two.

-- just another (another@engineer.com), October 28, 1999.

Big Dog --

Sorry, for the reasons I mentioned in reply to Don and Malcolm, I will have to pass on it.

Pinkrock --

Ouch. At least the person you talked to sounds like he might be a little better grounded than 'The Engineer'. (A little humor there, couldn't resist.)

Chris --

(And Pinkrock),

This is just one of the points that I think most serious folks with knowledge have attempted to make. There is a lot of uncertainty. In the power industry, (although see Malcolm's post above, which strikes me as a potential ray of sunshine. But not a 'happy face', Alfred E. Neumann, 'what, me worry', type of thing. Sounds like it is connected to reality.), in chemicals, medical equipment, etc. And the worst of it is that we all see only our couple of pieces of the puzzle. Malcolm is looking at his pieces and they connect together and are okay, and I am looking at six pieces that don't go with each other, and I don't have a clue what they would look like, and so on, then we all have to take a deep breath and 'guess' what everything else looks like based on the pieces that we can see. 'The Engineer' looks at his two or three pieces and they fit, lock together real tight, and says 'see, the whole picture is okay'. Malcolm says 'Well, mine fit together okay, but I don't know about the rest of the puzzle and I am missing a lot of pieces, so I don't know.' Flint looks at the pieces he sees and says the same. I look at the pieces I see and worry, possibly excessively, about all the others, because one or two of mine have bent corners, and the back is peeling up, and gee one of them looks like it belongs with some other puzzle.

I realize that analogy is suspect, but that one seems to me to have some congruence with reality.

-- just another (another@engineer.com), October 29, 1999.

Stan Heidrich --

Thanks for the reply. I state again, I am not so much worried about any one industry. I am worried about the chips out there and their potential to cause problems. Your point on the fuel industry is a good one. Thanks for the info.

-- just another (another@engineer.com), October 29, 1999.

'The Engineer' -- I will get this over short and sweet. I will apologize to you for the digs I made yesterday on this thread. The ones you got and the ones you didn't. This has bothered me all day, I would like to think I am a better person than that.

I will not respond to you further, though. You seem to be treating the discussion as a forum to score debating points, rather than have a useful discussion.

-- just another (another@engineer.com), October 29, 1999.

Big Dog --

Good point. A lot of discussion has focused on the power industry. I suspect that this is because they are such an obvious choke point. And I sure do hope that 'The Engineer' is correct. But even if he is right about his industry, what about the others?

Brooks --

I am going to let you and him fight this one out. May you have better luck, and a whole lot more patience than I.

Jim Worden --

Read you post and it was a good one. I followed the link to the Beach site and read the post there. Pretty good. His description of the memory types was more detailed than mine, and I would highly recommend it to the non-tech types. It was in plainer language than mine. You don't have to agree with everything he says, as there is some speculation in there, particularly toward the end. (I suspect that it is very difficult *not* to do this, once you begin to have the suspicion that that yellow substance dribbling down your leg is not lemonade. And the inconsistencies, and negative inferences, and instances of, as one of the posters put it, 'the dog not barking in the night', *will* cause that reaction.)

-- just another (another@engineer.com), October 29, 1999.

Mr. Brock --

Amen. At this point, the questions have become:

-- What do I know for a fact? (Normal answer, 'not near enough'.) -- What do I suspect? (Your mileage may vary, but probably 'A lot'.) -- What can I do in case of 1 or 2?

Which is where this forum comes in. One of the things I tried to point out is that it makes a difference if this is likely to be 3 days, a couple of weeks, a month, or a year. Particularly as to the infrastructure, of which power is only one component. (Jim W. also pointed this out on the previous post.)

BTW, would you by any chance be the 'brock' that Shakey was talking about?

-- just another (another@engineer.com), October 29, 1999.

Don Florence --

An astute analysis. The overseas thing is true. (Been there and seen that!) Ours is ****much**** better. Your point as to the main problem is also likely to be on the money. Catastrophic failure is possible, as this thread initially pointed out. But the cumulative failure mode is also possible, the 'avalanche' or 'snowball' effect.

I think this was touched on in one of the responses abve.

-- just another (another@engineer.com), October 29, 1999.

Bob Brock --

You and me both! But I suspect that the best we are going to get is 'The Engineer' who makes blanket assertions, and Malcolm, who is saying that what he knows about is good, but that he offers no guarantees about other stuff. (I know which one of these two *I'd* rather rely on.)

And the answer to the question about clocks on chips is that you can. It isn't necessarily easy. And it may not be possible to do it in such a way as to capture any interprocessor problems, or interface problems, since it will most likely have to be done in isolation. Possibly even only an emulation will be available.

-- just another (another@engineer.com), October 29, 1999.

I realize I haven't responded to everyone that posted an answer here. But it is late, I'm tired, I've got to get up and do it again tomorrow, and it is going to be a long day. I'd like to thank everyone who posted to this thread. (I didn't realize it would be this 'popular' a topic. But looking at some other threads, it seems to have hit a nerve, and probably needed to be re-examined.)

I'm going to call it a night, and I may not get back to this forum for a couple of days, due to the press of work. And I'd like to add a couple of items.

First, I don't claim to be the absolute last word on this subject. There is a thread started by Sluggy who appears to be working with embeddeds currently. I do not agree with everything he says. I suspect that he is over-generalizing. But that is also one of the points I tried to bring out. *NONE* of us has a real clear overview of the picture. Take a look at Flint's response on that thread. It is a cogent statement of the pitfalls related to this subject. And he is also right that no outcome between 'absolutely nothing' to 'total meltdown' would be surprising.

Some of one's outlook depends on perspective. One looks at what they have been working on recently and it is okay, and the pace of change is rapid, and therefore everything is ok. Others look at it and see stuff that they did on systems twenty years ago and it is still there and still in use, and they go 'tilt'. Others see something in between. And none of us can say with certainty that we are right.

I didn't expect when I posted this that there would be any guarantees. I didn't get any. I got information from this and other threads that may make me sleep better, for knowing that maybe it isn't as dismal as I had thought. I got other information that may keep me up a couple of nights worrying.

I sure hope that there isn't anybody taking anything I have posted as gospel. In case anyone did, here is another disclaimer. *I don't know.* There, I think that is clear enough for anyone.

What I wanted to try to do was to give some of the non-engineers an opportunity to understand a little better what an embedded system is, how they work, what some of the pitfalls may be, and why I worry about it. (In case they want something more to worry about.) But I want to emphasize that it might be a case of 'garbage in, gospel out'. It has been 5 years since I was last involved directly in coding an embedded system. (The place I work now I am only indirectly involved in the support aspect, enabling all of those embedded developers.) I am not sanguine about the prospects for the embedded stuff there, and I will admit it. I think they have overlooked some stuff, and I think it will bite them. There are other aspects of the process which I think are vulnerable, but they aren't 'customer visible' except in the sense that the customer is going to have a problem getting the errors that can be seen fixed.

But there are systems out there I worked on. I have racked my brain, and I *think* there is only one of them that could pose an extreme safety hazard, from an operational standpoint. The others are mostly things that will just be 'lived without'. And it was always mandatory to cause equipment to 'fail safe', so that a computer failure didn't do any damage to anything. But we didn't design them for this. In those cases where it was considered the answer was always, 'It will be replaced long before then.' But they weren't replaced. And now we sit here and worry and wonder. (And yes, I do feel a little guilty about it. We could have said that under no circumstances would we send out something that might have an undetermined failure mode in it. But we all liked working and collecting a paycheck. And it was easier to say 'it'll fall through the case statement, and the default case is a fail safe.' And continue. And now we find out if we were right.)

I guess the bottom line is that if this thread made people think, then it was worth doing, and worth the couple hours sleep I missed responding to folks. If it helped anybody clarify their thinking, on *either* side, it was worth doing. But it is *only* 'just another' input. You have to do your own processing. I can't tell you what to think about it. Flint can't. (Sorry about picking on you Flint, but you are one of the more respected posters about these sorts of issues.) Other folks will tell you what to think. Try to look past that and consider what they do say. It's data. And you cannot make up your mind without data.

There was one thing on here that I will agree with 100%. Brock's statement that it is 60 some odd days out and whatever is fixed is fixed, and whatever isn't is not likely to change state in the next 2 months. All any of us can do is make the most educated guess about what is and what isn't that we are capable of, and make our decisions based on that.

This has been a bit rambling, (probably should have warned you beforehand), but these were things I thought needed saying, and I doubt that I will be back on here for a week or so. Good luck all, and good night.

-- just another (another@engineer.com), October 29, 1999.

Don, You are correct. However since in many third world countries electricity is so unreliable and does go down for reasons having nothing to do with Y2K it would seem logical that the people living there have come up with ways to cope for the last X number of years. Therefore an outage due to Y2K wouldnt be a big change unless it was very prolonged.

As for the second part of your question: It depends on how you define outage. Outages happen all the time due to lightning, storms, etc. I was out of power for an hour yesterday due to a tree branch in the line (high winds). If you are talking about wide spread outages (grid) its usually the result of large bulk transmission lines being put out of service due to physical damage. Sometimes there is whats called cascading outages. Usually its a combination of the grid being stressed to the maximum (which wont occur over Y2K) and equipment failures.

Bob,

What you are looking for you wont find here. We dont test every chip. We test (in example) a relay with a chip in it. We dont test each identical relay which all have the same chips. Again the problem is that there is the belief that everything has a chip in it, all chips have dates and all (or a significant number ) will fail. That is wrong. And, at the risk of repeating myself ad nauseam, the dates dont do anything except tell you the date. Not much is controlled by mainframes by the way.

Jim,

Beechs basic premise is wrong. Its kind of like saying lets assume the earth is flat and go on from there. This came around several years ago in various guises. Some people thought that chips were programmed with clocks set to the local time zone. I.e. chips made in Indonesia were set to Indonesian time, chips made in New Mexico were set to Mountain Time, etc. Others thought the clocks were set to GMT time. One writer called them secret clocks. I dont know where you live but if you are near a good community college or any college print out what Beech wrote and wonder over. Find a professor or grad student in the EE or computer department and ask them to read it. Maybe buy them a cup of coffee as a bribe. I think theyll agree with my assessment.

As for protecting ourselves: The same way we always do. Various relays are set to trip breakers on the loss of tie lines due to over or under frequency, voltage collapse, etc. You have to always worry about that regardless of any specific date and time.

Preparing: No you didnt Get It. You only think you Got It. My guess is youll never Get It. You really dont understand embedded systems at all. How long have I known about Embedded systems? Well since Ive been an engineer for 26+ years how long do you think?

Old electronics man: Yes it is hogwash and if you dont know that you dont know as much as you think.

Bob: True I dont program the chips, at least as you mean it. But I do use them. And I have talked to people who have programmed the ones we use in the equipment that we brought. So I have very specific knowledge about the equipment not just general blah, blah, blah. Do remediation? None was needed in my specific area. The area where it was needed (some SCADA, etc) was very small. Basically replaced a few chips in non-critical hardware.

K. Stevens: Actually most of them are from Motorola.

-- The Engineer (The Engineer@tech.com), October 29, 1999.

just another-- no, I am a different Bob Brock, at least everyone says so. I'm not a DOOMER, I'am a "DON'T KNOWER" and a "DON'T TAKE A CHANCER" If you go down my basement, you'd swear I was a DOOMER. I don't have anything useful to contribute to the discussion. I've read all the items on Gary North's site (The BEST site for information. You don't have to read his opinons if you don't want to, but for DATA, his is the best! I bet when this is over, he will have saved more lives than anyone since Jonas Salk.) There is very limited information on chip PROGRAMMING, so I really appreciate your input and information.

The oil industry has found 20% failure rate in chips that were ok'd by the vendor.

earlier this year on North's site there was an e-mail from a large farming complex in California, if I remember right. They tested the chips in their irrigation system. There were identical model valves that had one check out and one fail. So much for like-kind testing.

ps Mr Cory Hamasaki is Great reading for mainframe reading. It's like reading a Jack London novel. My wife hates his style...go figure.

The Engineer--

I read the Beach Bug theory. It was a theory. Theories are worth concidering. His later Responses and rebuttals were backed-up by some VERY impressive people. Mr. Beach stated all along that it was a theory.

I do appreciate your input and I sure hope you are right...but I for one will not bet my families safety on hope.

good night Mrs Calabash...whereever you are.

-- bob brock (rbrock401k@aol.com), October 29, 1999.

This is most amazing. Most enjoyable, the information is like finding water in the desert. It took some digging to find people actually talking about the issues that have concerned me from the get- go. I am not an Engineer, I majored in Economics, then moved into IS. My company is Compliant, whatever that means. I must be an idiot thou--. The PLC's and Microcontrollers I have on my manufacturing equipment have been verified based on the Vendors "o.k. - its compliant". Granted my company is small ($10M), with a very small IS staff (I am only 5'2" 105lbs)LOL -- sorry I am finding very little to humor me these days --I never thought the vendors of my chips my lie? Does anyone think that 35 miles north of San Francisco is far enough away. San Quentin is a concern thats come up in the last couple of days, thats only 25 miles down 101. YIKES--------anyway this has been some great reading, Good luck to all the contributers --Karla

-- MIS (KarlaCALIF@aol.com), October 30, 1999.

just another-- I have been reading this thread all evening, and I would just like to say---you have accomplished successfully what you originally intended --giving non-technical, non-engineers an understanding of the imense scope and complexity of embedded systems. I applaud you and thank you for the obvious inordinate amount of time you have invested. Your sincerity to your original objective is extemely evident in your conciencious responses to the "non-engineers". I have enjoyed almost all of the posters. I can well understand your hesitation with future correspondence with "THE 'Omniscient' Engineer", although his last few comments seemed to be a little less condescending, he still needs to learn some manners "or is it too late to teach the old dog some new tricks?" Anyway, thats not what I am here for, just wanted to say thanks. ps, don't laugh but can someone please tell me what a polly is? I am a fulltime + MIS Manager and a fulltime + Mom to two beautiful daughters, I don't get alot of free time to participate, but I must say this hooked me. ;) --Karla

-- MIS (KarlaCALIF@aol.com), October 30, 1999.

Mis a Polly is a "pollyanna"....someone that can be chin deep in doodoo and is thrilled that she is so tall. looks at the bright side of everything, refuses to acknowledge the bad. it sounds better than it is.

-- bob brock (rbrock401k@aol.com), October 30, 1999.

We have a real fear. Previously we had thought Cascadia would do better than some because of the hydro-electric capacity. We know they can "go manual." We took care of some of the builders & engineers & maintenance pioneers on that system. But now this arrogant troll Engineer swags and brags about how incomplete, unthorough, and overconfident they've been. Makes us sick, retch to a 10 personally, locally. Not to mention their systemic PR and that big sale to Scotland.

Time to print this thread and personally bring it to the Mayor of Portland.

-- Ashton & Leska in Cascadia (allaha@earthlink.net), October 30, 1999.

bob brock-- Aaaahhhh!! Thanks for the clarification. Just a side-bar, read Paula Gordons "White Paper" last night. Very interesting reading, but in section three there is a part where she is talking about personal preparedness, and the common sense or lack there of she discusses in part six could also apply to her. In one paragraph (section 3), she talks about the need for "~gardening and other steps to increase food supplies", and the next paragraph, she defers to the American Red Cross check list which suggests only a few days to a couple of weeks supply of food and water. I think in general, her paper address alot of issues and concerns that should have been taken more serious, but under the remaining time constraints and at this late date common sense would dictate its to late to do much but prepare. (alot) Someone out there will probably get a kick out this. Last night on C- Span a group of congress men/women were discussing Y2k with Willmenson and someone from the GAO, one Senator actually suggested a 3 person committee should be set-up to communicate with the public anything that may occur during the roll-over. His suggestion for the main spokesperson "a Walter Cronkite type, that the American people will trust" ---- I had nightmares the rest of the night (at least he didn't suggest clinton). --Karla

-- MIS (KarlaCALIF@aol.com), October 30, 1999.

Karla -- yes, the extent of their DGI is nightmare-inducing, eh? Paula Gordon posts on this Forum. She is on record as saying 9.5 if TPTB continue along their present course. Which they are blindly doing, tra la la la la.

Welcome to our Twilight Zone, and stay on our News Coaster for the ride. There are enough senate reports, GAO findings, insider info, personal observations, and technical discussions to rev your alarm engine to high speed. The vast majority of sleeping weeples out there has absolutely no inkling ...

-- Ashton & Leska in Cascadia (allaha@earthlink.net), October 30, 1999.

Ashton and Leska--

;-) Thanks, I intend to. I would much rather be on this ride discussing these concerns than the "Twilight Zone" called complacency that I interact in everyday. I have given up trying to warn people of the possiblities. Its nice to find people who are (as much or more so) aware and concerned about the possiblities as I am.

-- MIS (KarlaCALIF@aol.com), October 30, 1999.

MIS --

Glad it was helpful. (BTW, ROFL at your description of a 'small' MIS dept. Don't have all that much to laugh at sometimes, and I, for one, will take what I can get. Thanks!)

ALL --

I hadn't intended to get on here today. I am working from home and have a ton of stuff to get done. But decided to take a short break from 'lift that barge, tote that bale' and see what was new and came across a thread that is an absolute, positive, MUST READ for anyone who wants insight into the whole Y2K thing. And decided that it was a good enough thread that I needed to bring it to the attention of the folks who posted here, as it bears on some of what was discussed here.

PLEASE READ IEEE Y2K CHAIRMAN'S PERSONAL, PESSIMISTIC TAKE ON Y2K AND YOURDON'S END GAME ANALYSIS by Dale Way (Chair of the IEEE Committee) posted by Roleigh Martin. The essay is LOOONG, some of it is technical, but it IS readable, by any, including laymen who take the time to puzzle through it, and it is an EXCELLENT analysis.

Thanks

-- just another (another@engineer.com), October 30, 1999.

Dear MIS (KarlaCALIF@aol.com),

I think you missed the intent of what I was saying concerning stocking up on food and water and my reference to the Red Cross. Please note the words: "AT A MINIMUM" and "AT LEAST". In citing what the Red Cross was saying at the time (in February), I was trying to communicate to people who do not get Y2K and who had no idea that the Red Cross had made such recommendations, that indeed the Red Cross was concerned at some level and had made some recommendations, minimal though those recommendations were, and that making minimal preparations was at least a beginning. Here is the entire section in context (taken from Part 3 of my White Paper posted in February of 1999). The only difference in the text is that I have put the two key phases in all caps: AT A MINIMUM and AT LEAST. Also, I should note, when I wrote the words "severe winter storm", I had in mind the most severe ice storms of the type that stopped life in its tracks for over a month in the Northeast and Canada.

The following paragraph is from Part 3 (posted 2/25/99) of "A Call to Action: National and Global Implications of the Year 2000 and Embedded Systems Crisis" by Paula Gordon at http://www.gwu.edu/~y2k/keypeople/gordon

"~ The Need for the Gradual Stocking up on Water and Non-Perishable Food Supplies. AT A MINIMUM, the public should be urged to gradually stock up on water and non-perishable foodstuffs needed for a period of a few days to a few weeks. (The American Red Cross advises stocking supplies to last a few days to a week. See the latest material on Y2K preparations being disseminated by the American Red Cross on the American Red Cross website. The Red Cross also advises having an "ample supply" of prescription medicine on hand. In areas where worse case winter storms have been known to occur, inventories of supplies should be AT LEAST in keeping with FEMA's minimum suggested guidelines for preparing for severe winter storms." (End of quote from White Paper) It is October 30. There is still time for individuals, families, and communities to prepare. It will take alot of urging and a major increase in awareness, but it is possible. Preparedness efforts will need to go on after the first of the year as well.

Preparedness efforts are increasing around the country to the help people understand not only the unknowns involved in this situation, but also the steps that they can take to help themselves, their families, and their communities to meet the challenges ahead.

There is a list of references and resources that includes a section on preparedness on my website. It includes The Cassandra Project, the UTNE reader, Sally Strackbein's Y2K Kitchen (http://www.y2kkitchen.com) and a variety of other resources.

A realvideo website can also be accessed from my website. It includes some excellent presentations on community preparedness.

My October rating and comments regarding the likely impacts of Y2K are at http://www.russkelly.com and also at my website in a new section entitled: Comments, Essays, and Op-ed Pieces. Observations about preparedness efforts are offered there as well.

There will be a free program on community preparedness at on November 23 at GW University, a part of a conference series beginning November 10. See the announcements page on my GW website cited above for details.

I have been focused primarily on trying to help people in roles of public responsibility, including the media, increase their understanding of the nature and scope of the crisis and what might be done about it. I have especially focused attention on minimizing the likelihood of disasters involving the highest risk, highest hazard systems, plants, sites, refineries, pipelines, etc., in part because so few seem to be aware of the dangers there and in part because disasters of such magnitude could make infrastructure disruptions the least of our concerns. A major purpose of the November Conference Series at GW is to focus on such overlooked high risk, high hazard areas of concern. (See the detailed announcement at my GW website). I have indicated that preparedness efforts need to go on simultaneously.

I am attaching two recent comments that I have submitted to TB2000 to give you an idea of more recent statements that I have made regarding preparedness.

I hope these give you a clearer idea of where I am coming from on these matters.

ITEM 1

October 24, 1999 submission http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=001dd7

"FEMA's Project Impact program is aimed at building disaster resistant communities: Project Impact could be quickly adapted to assist communities prepare for Y2K. This multi-million dollar investment could help in preparing for and minimizing impacts of Y2K. Why isn't this national asset being used in this way?

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Project Impact is a promising program of the Federal Emergency Management Agency. The purpose of Project Impact programs is to build "disaster resistant communities". Some excellent guidance materials have been developed and are freely available through FEMA. The materials are filled with useful ideas. The program could be quickly adapted to serve as models to assist communities all over the country to prepare for Y2K. Project Impact is holding a Summit in Washington, DC, December 12 to December 16. Representatives of over 116 programs from throughout the nation will likely be there. Last year over 600 people attended. If last year is a guide, all of the FEMA regional directors will also be present. There were only rare mentions of Y2K at last year's Project Impact Summit. In the 14 page Summit overview for this year's meeting, not one mention is made of Y2K. Surely it will be a major embarrassment to FEMA if it turns out that FEMA never alerted its Project Impact grantees to the threats and challenges of Y2K and the need to prepare.

How ironic that those who have been participants in such a promising program have not been encouraged to redirect their efforts to preparing for Y2K. At best, they could be serving as models for organizing for community preparedness. At the very least, they could help their own community to prepare.

Information about the Summit can be found at http://www.fema.gov/impact/summit99.

In a letter from James Lee Witt quoted in the preliminary program for the Summit, he states "Together we are building safer, stronger communities before disasters strike." Why is the obvious relevance to preparing for Y2K being missed?

It would not be that hard to maximize the sunk investment that the nation has already put into the Project Impact program. What has happened to American ingenuity, creativity, and just plain commonsense?"

-- Paula Gordon (pgordon@erols.com), October 24, 1999

************************************************************* ITEM 2

October 29, 1999 submission that I sent in to a thread at http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=001fe7 (By the way this thread is filled with lots of references that might be of help to DGI's and others who are not making preparations):

"I wrote my White Paper on the Y2K and Embedded Systems Crisis for DGI's in all walks of life from the President and Vice President, to Mr. Koskinen, media folk, and all of the rest of us. I hope it might help your sister understand why it is that the government has not been doing its job when it comes to Y2K, and what needs to happen in the remaining weeks (Part 5 focuses on that). Some points that I have tried to bring home are the following: ~The technological aspects of the challenges and threats posed by Y2K and embedded systems are apparently not as yet comprehensible to a majority of those in leadership roles in the Federal government including the President, the Vice President, and Mr. Koskinen;

~ The economy and narrow political considerations seem to be uppermost in the minds of these individuals;

~ They are apparently afraid to tell the public what they do know for fear this would panic the public; they are assuming that the public cannot handle bad news and rise to the occasion;

~ Many of those in key positions of responsibility apparently have decided that the public are not worthy of being treated as adult citizens of a free society;

~ They have made a colossal error in judgment thinking that there will be less public panic if the majority of the citizens of this nation go into the new year without having made any preparations that could have helped them and their families meet any challenges they might face.

~ Unless the current approach changes radically, there are going to be a lot of angry people around when they realize how poorly the Federal government has served them in failing to warn them, inform them, encourage them, and assist them in making personal, family, and community preparations.

~ Unless, major initiatives are launched in the remaining weeks, there are going to be even more angry people around when they realize that how little the Federal government did to prevent the technological disasters that are likely to occur;

~ To date, a majority of key government leaders have shown an astounding lack a sense of responsibility and integrity when it comes acquainting themselves with the threats and challenges we are facing and, once armed with that understanding, fulfilling their obligation to serve in the public interest and to act in ways that preserve the General Welfare and the security of the nation.

They are asleep on the job. Citizens have a responsibility to help wake them up. Communicating your concerns to elected and non-elected public officials is one way. Communicating your concerns to the media is another. Communicating your concerns to others you know and others in your community is another. Lots of resources are out there to drawn upon and help you make up for lost time. A list of selected references and resources, including resources on preparedness is available at my website. All six parts of the White Paper can be found at that website: http://www.gwu/edu/~y2k/keypeople/gordon. A growing number of video programs, panels, and presentations on Y2K and embedded systems can be found on the realvideo website that can be linked to from the above URL. A panel program on community preparedness is included among those videos. One of the most helpful presentations on that community preparedness panel tape is a discussion by Phillip Bogdonoff of the Center for Y2K & Society concerning the various stages that most individuals have to work through in coming to grips with the implications of the seriousness of the situation that they find themselves in when they finally "Get It"."

-- Paula Gordon (pgordon@erols.com), October 30, 1999.

Paula--

Thank you, I guess I must be somewhat sensitive when it comes to personal preparedness, as you say --many people have not even seen the Red Cross release, I personal have printed out 100's and passed them on to people who have not seen or heard (or have access to) their guidelines. I have also added my own opinion while handing these out, "that they should be considered minimum preperations." I have read your entire paper and commend you for the work you have done. Awareness at all levels is critical, I strongly agree (and like you feel there is a severe lack of leadership from the president on down) Do you know where I can go to find out more information/explanation regarding your (the) scale on impact, your range is currently 4.5 to 9.5, I would be extemely interested in the details for these outcomes. Again, I apologize and thank you for the additional information and clarification. Karla--

-- MIS (KarlaCALIF@aol.com), October 30, 1999.

Paula-- www.russkelly.com (the link for your ratings?)

-- MIS (KarlaCALIF@aol.com), October 30, 1999.

Karla,

Thanks very much for your response and kind words. I am glad that what I wrote helped clarify recommendations I have made concerning preparedness and what the government should be doing about it.

The impact rating scale is explained in Part 1 of my White Paper at http://www.gwu.edu/~y2k/keypeople/gordon (Click on Part 1). My rating comments can also be found at that same website. Click on "Comments, Essays, and Op-Ed Pieces".

In addition, my rating comments along with those of around 20 others can be found at http://www.russkelly.com.

Regards,

-- Paula Gordon (pgordon@erols.com), October 30, 1999.

to The Engineer

I believe you were probably talking about the programmer of you EMS system that said your relays were ok.

the field programmer of your EMS did not write the main program of the EMS company. He filled in what points you wanted to monitor or control. It's a little like you programming your phone for speed dial...your not writting the program. The EMS company did not write the code in the chips, even if the relays/actuators/controllers were of their manufacture. Same relays may not contain chips from same manufacturer even though they do the same function. anybody...correct me if I am wrong.

by the way....if I check my pc and all my programs and they are ok, am I safe? Do I need to up-grade my monitor, keyboard and mouse? Could this be the obvious thing that we all miss?

-- bob brock (rbrock401k@aol.com), October 30, 1999.

Just another

Thanks for your efforts. You have put alot of time on this thread.

And here is the link to Mr Ways article (thread)

 IEEE Y2K Chairman's Personal, Pessimistic Take on Y2K and Yourdon's End Game Paper

-- Brian (imager@home.com), October 30, 1999.

Brian --

Thanks for posting the link. Also, if anybody here *hasn't* read it, the article that is being critiqued is *also* a must read. (That is, Ed Yourdon's End Game Analysis.)

Whether you agree or disagree with the two, both are absolutely top notch evaluations of how these two individuals view the coming situation. For those who might be unaware of the credentials of the individuals mentioned, Dale Way is chair of the IEEE Y2K committee, and for the *really* technically impaired, IEEE is the Institure of Electrical and Electronic Engineering, the preeminent association in the field; Ed Yourdon is one of the early pioneering theorists in the software engineering field, has written a ton of books (I think literally, my stack adds up to about 25 pounds, and I don't have all of them), and in those design and analysis circles I used to move in, it was widely assumed that he didn't walk on water *only* in order not to embarass those who couldn't. (In other words, in my humble opinion, both of these gentlemen are worthy of being listened to as they have valuable contributions to make.)

Bob Brock --

You know, it occurs to me that we are maybe being a little hard on 'The Engineer'. Unless I am mistaken, his responsibility is the hardware system that produces power, and it seems to me that he probably has a different mind-set than software engineers. I mean, think about it. What hardware engineer would ever *CONCEIVE* of rolling out an 'alpha' version of a transmission system? A 'beta' version of a generator? What I am pointing out is that he operates in a *very* disciplined field. And probably has trouble envisioning just what a load of utter, out and out *CRAP* gets shipped every day by the 'software engineering profession'. (Which is almost an oxymoron once you put that 'profession' on it.)

For years and years people in our field have been cowboys, undisciplined, and unaccountable. We tout vaporware, ship stuff that is wrong. (I know of an instance of a division of a Fortune 500 company that shipped something to a major customer that wouldn't even load. When the customer caught it, the installation team sent back for a clean version. Which also wouldn't load, but for different reasons. And if an F500 company can do this, what about all the little two-by-four startups out there?)

This line of thinking has given me a whole different perspective on the though process which *may* lie behind the harsh opinions of some of the posters. I need to think it through a bit more, but I may post a thread dealing with it later.

And by the way, I don't think that your keyboard, or mouse need to be replaced. Now your monitor, that may be a different story. (Just kidding. I think. That is, I hope I am kidding. Does yours have memory in it for storing configuration and set up data? Surely it doesn't have any date dependencies. But you do have a certificate of compliance from the vendor? Wouldn't that be funny, if we fixed everything else in the world, and then couldn't access it because all of the monitors crapped out?)

-- just another (another@engineer.com), October 30, 1999.

Thank you all for this thread. It reminds me I am not alone in my preps. After all the reading, its still a big WE DON'T KNOW.

The ENGINEER is right. Power plants without digital conntrols will work. But more modern plants with digital safety systems could be shut down by one malfunction. I remember reading about one chip in the shaft of a generator that measured vibration; it failed on testing and it took them a month to find it.

My big concern is oil. U.S. oil wells are said to be going FOF with an expected failure rate of 10%-20%. Those chips hundreds of feet under ground or thousands of feet below sea level will take weeks to replace. Foreign oil (50% of our usage) is questionable because we don't know the level of their well remediation and the remediation of ships and ports is also a big question. Our national supply of crude is good for 60 days.

There is going to be an oil shortage. Back in '74 oil production dropped 2% and we saw gas prices triple. What will a 10% drop do? Will the Feds limit gasolene production to produce diesel for trains and trucks? How will we all get to work?

-- John Littmann (JTL9700@JUNO.COM), October 31, 1999.

I don't know about you John, but I for one am ready for a vacation ;-) Just Another LOL-- "a life without monitors" --oh my! Bob -- Thanks for the link to IEEE Y2K Chairman's Personal, Pessimistic Take on Y2K and Yourdon's End Game Paper.

-- MIS (KarlaCALIF@aol.com), October 31, 1999.

Taking these from the bottom up:

John Littmann: I think (but admit Im not sure) that the story about the chip on the shaft in a Y2K urban legend. I tried to trace it down and was very unsuccessful. I also think the stories about chips 100s of feet down are another Urban Myth. Chips fail for all sorts of reasons. Electronic gear tends to follow the classic bathtub curve failure rate. So any chips buried would have a chance of failing regardless of Y2K. That would be increased if they were in a hostile environment. The idea here seems to be that the oil rigs (whatever) are vitally dependent on these chips yet they are a) in an inaccessible place and b) wont fail for any reason except Y2K. I find that a bit much. Its outside of my field so I cant say its definitely not true, but doesnt seem to make much sense.

Just Another: Nope Not a hardware engineer, in terms of computer hardware. Power Engineer. I really work on the systems in the field. Dirt under the fingernails type engineer.

Bob Brock: No the EMS system and the relays are entirely different and separate systems. I really wouldnt even call the relays a system. I would have to go into a long long article. Hopefully Malcolm will cover it in 101 but the way you are thinking it operates is incorrect. Its true that we dont write the program. However the relay software is set by us. And its the setting that are important. As Ive said before time doesnt play a part. Timing does Each position has its own relaying and that relaying is set to protect that position. EMS is more of an AGC (Automatic Generation Control). At lower voltages it can include tap changing and such. But we do that by command and not automatically. One of the hard things to understand is that its not really on auto-pilot. There is a lot more human interaction on a continual basis then the people on this forum seem to know or understand.

Beechs theory isnt worth considering. Look at it this way. No one really knows what gravity is. There are lots of theories but you do know what the effects are if you jump off of a 10 story building. And you can calculate its effects using both Newtons Laws and Einsteins Relativity equations. Ditto no one actually knows what electricity is. But it can be used. The people at Intel, etc. know how their chips work and dont work and whats programmed into them. For Beech to come out and talk about secondary clocks is a bit much. Especially when no one from the industry has come out and said: Hey you know that guy is right!. Now it could be that they missed all of this but I dont think so.

-- The Engineer (The Engineer@tech.com), November 01, 1999.

I also think the stories about chips 100s of feet down are another Urban Myth. Chips fail for all sorts of reasons. Electronic gear tends to follow the classic bathtub curve failure rate. So any chips buried would have a chance of failing regardless of Y2K. That would be increased if they were in a hostile environment. The idea here seems to be that the oil rigs (whatever) are vitally dependent on these chips yet they are a) in an inaccessible place and b) wont fail for any reason except Y2K. I find that a bit much. Its outside of my field so I cant say its definitely not true, but doesnt seem to make much sense.

The Engineer,

Here's the earliest Web page I'm aware of that talked about the location issue:

[Fair Use: For Educational/Research Purposes Only]

[added bold emphasis mine]

Archive

April 1998 Vol. 219 No. 4

Feature Article

Will the millennium bug give your operations the flu?

Don't take the head-in-the-sand approach toward potential computer strangling of production operations. Time-contingent process controllers must be evaluated for year 2000 date stamp limitations and their implications for safety, the environment and operations

Scott M. Shemwell, Jerry Dake and Bruce Friedman, MCI Systemhouse, Houston, Texas

Human history is replete with mystical and religious concerns over the end of a millennium. Armageddon or end-of-the-world scenarios are typical refrains. This time, oil and gas producers may face a more identifiable plague. Then again, Jan. 1, 2000, may come uneventfully  as has every thousand-year transition of the past.

For more than 15 years, the oil and gas industry has expended a massive effort to re-invent itself. We all know that none of our firms would be competitive in today's market, if we had not made these hard decisions. A linchpin of the industry's success has been the reduction of the corporate cost structure through the use of technology and process re-engineering, much of it computerized. All of this work is potentially at risk, if serious loss of production is sustained as a result of unplanned computer shutdowns in many segments of the business, all at the same time.

THE MILLENNIUM BUG

As we close on the first 100 years of the "information age," we are faced with a legacy from the medieval computer past. In the computing dark ages, processing power, memory and hard disk space were an expensive premium. Like the wizard Merlin, programmers of that bygone era concocted software brews, the recipes of which now are, more often than not, non-existing. They certainly did not take one important fact into consideration.

No one expected that some legacy software, with roots often over 30 years old, still would be in general use today. Further, as these recipes or programming techniques were taught to modern-day wizards, they, too, adopted the same incantations. Therefore, even new software programs may have the same limitations.

How the problem started. Today, we live in a world in which computer software is fundamental to our very way of life. Computers are everywhere  from mighty mainframes, high-performance workstations and PCs, to games, toys and even automobiles and household appliances. Many software programs driving our business functions have one thing in common  limitations of the past dictated that the variable calendar year be represented by two digits instead of four, e.g., 1966 would be expressed as 66 and 1998 would be 98. This was an efficient method and did not present any problems initially. This date stamp limitation is the so-called "millennium bug."

A simple example. As one example, consider a simple problem. An oil and gas market researcher is interested in the buying patterns of his forecourt customers. He commissions a survey and asks 100,000 individuals a series of questions, one of which is their date of birth. In his analysis, he correlates age to a number of other variables, builds a profile of his customers and uses this profile as part of his next-generation product planning. Sound familiar? Well it should, because it happens every day.

What if our hypothetical researcher conducts the same survey in January 2000? If his statistical software calculated the year by the last two digits (00), he may find three types of error:

1. He will discover that individuals born in 1960 are not 40 years old, but minus 60, e.g., 00- 60= - 60. The astute researcher will see this problem immediately and adjust accordingly to this inconvenience.

2. Any calculations involving the age of respondents such as "percentage of population over 30 years old" will be incorrect. This mistake may be more difficult to find and rectify, because age may be a variable in several processes. However, this is still largely a further inconvenience.

3. Age, or calculations made from age, may be the basis for more sophisticated analyses errors that may not be readily apparent to the researcher, such as what might occur with matrix algebra. When multiplied by 100,000 samples, this error may impact the validity of the analysis seriously. Business decisions made on the basis of these analyses are likely to be inaccurate and fail.

A serious problem. Our example is straightforward, relatively simple, and most errors can be detected and compensated for easily. The real world is seldom simple, and the stakes may be a good deal higher. What if, instead of an off-line market research project, our system was one or all of the thousands of embedded or "computer on a chip" process controllers on an offshore platform, Supervisory Control and Data Acquisition (SCADA) system or distribution pipeline? What is the impact of these three error types cascading throughout multiple- intertwined and mutually dependent on-line systems?

An offshore platform may have 10,000 or more embedded silicon chips governing all automated and even some manual processes. Many of these systems are subsurface or underwater and physically difficult to access.

PROCESS CONTROLLER CONCERNS

Unlike the software of a marketing system, the embedded logic on a silicon chip is entombed deep in the system and not easily ascertained. Any given Distributed Control System (DCS) or Process Logic Controller (PLC) computer board has many chips, and their interdependencies on each other, and on other system components, make them difficult to analyze and repair, Fig. 1.

Methods for analyzing this equipment are only now emerging. Compliance information coming from manufacturers has been sketchy and sometimes inaccurate. In some cases, the chips are no longer made. In others, the controller is manufactured in such a way that the entire unit must be replaced. Upgraded chips and new controllers also would have to be tested to ensure that their insertion will not impact drilling and production processes negatively. Some studies suggest that there may not be enough manufacturing capacity to just replace all affected chips in less than two years.

Few organizations have recognized the full potential for possible failure in embedded systems. Moreover, the supply of talent qualified to identify and correct these problems is being consumed quickly by other year 2000 projects. The longer that production managers wait, the less the likelihood that they will be able to affect the outcome pragmatically.

It is estimated that the average oil and gas firm, starting today, can expect to remediate less than 30% of the overall potential failure points in the production environment. This reality shifts the focus of the solution away from trying to fix the problem, to planning strategies that would minimize potential damage and mitigate potential safety hazards.

Some systems are experiencing failures, already. In a recent case, that resulted in litigation, a point-of-sale retail system installed in 1996 rejected credit cards with year 2000 expiration dates. Other lawsuits are likely. The legal profession expects to earn billions of dollars contesting millennium bug failures. The Securities and Exchange Commission (SEC) also is getting involved in this crux, requiring corporations to establish financial reserves to cover year 2000 exposure.

CONTINGENCY PLANNING

The recognition of issues surrounding embedded systems is a relatively late entrant into the year 2000 discussion. Companies only now are becoming aware that the most likely threat to the revenue stream, environment and safety is more likely to come from an offshore platform or refinery, than from a mainframe accounting system.

The reasons for this oversight are straightforward. The year 2000 problem has been characterized as an information technology problem and delegated to each organization's Information Services (IS) department. However, IS departments typically do not manage on-line, process control systems. The embedded systems issue is a process or business problem affecting all types of "intelligent" equipment throughout all business units, not just those computer systems for which IS departments are accountable.

Consultants and corporate IS departments have developed methodologies and automated software tools to address year 2000 issues in most segments of the computer industry. Some components of these approaches are applicable to embedded systems. However, there are three major issues that oil field asset managers face that are not as relevant for IS:

1. Process controllers and associated "intelligent" devices are integral components of real-time systems, Fig 2.

2. These systems may have component parts that are not the property of the operator, such as rented compressors.

3. IS personnel may not understand the process linkages and cannot take the computer systems down for maintenance, e.g., office computer systems are often taken off-line late Saturday night for maintenance and upgrades, with minimal business disruption.

Addressing these issues will require a cross-functional team. It should comprise individuals who are knowledgeable about the engineering processes in question and their relationship to other processes, the equipment involved and its level of automation, process control systems, and year 2000 hardware and software issues. Reporting to an oversight committee and the executive sponsor, this team will inventory the functional processes and control systems associated with those operations. It will then contact the control system manufacturer (or access commercially available databases) to attempt to ascertain year 2000 compliance for the devices in question.

With this baseline assessment, management can determine which systems are mission-critical from the standpoint of safety, environment and business operations. Systems that are deemed to be critical will require action plans to ensure that those processes are dealt with appropriately. This may require switching to manual operation or a planned shutdown. Plans also can be implemented for systems that are determined to be non-mission-critical but may malfunction, too.

Large organizations should prototype each process, so that this knowledge can be distributed to global operations in a cost- effective, timely manner. Multiple parallel teams can be used as required, since it is unlikely that a single team can physically assess all systems in less than 24 months. Moreover, local engineering knowledge will be required, because many processes, while similar, are not exactly the same worldwide.

Some remediation can be accomplished during planned maintenance. But operators must expect that due to work volumes and time shortages, the contingency planning process probably will drive millennium bug resolution. Complex systems that include equipment belonging to multiple organizations further complicate the problem, because it is important to completely understand what signal is sent by each electronic device. For example, a smelting plant in New Zealand lost several months of production, because one of its controllers did not recognize the leap year and shut down when it received an electronic signal that was different than the expected date, March 1.

Computer chips are becoming ubiquitous, with over 7 billion manufactured in 1996, alone. The millennium bug not only can infect production processes, but also every on- and off-line process in the oil and gas value chain, from seismic acquisition to the pumps at the gas station.

TIME TO ACT NOW

The year 2000 software problem is real. Many computer industry pundits estimate that fixing the problem will require hundreds of billions of dollars, and a few place that figure substantially higher. This is an important issue, but it need not be Armageddon. As with our market research example, many failures will be just an inconvenience; others may be more serious. There are many uncertainties, but what we do know is that failures in process control and monitoring systems can shut down facilities, damage the environment and jeopardize safety. Management's fiduciary responsibility to corporate stakeholders suggests that we develop an understanding of our situation, initiate a remediation strategy with contingency plans, and implement those plans that are relevant to our specific situation.

On a Friday night less than two years from now, a tsunami will build in the Pacific and roll westward through all major hydrocarbon producing fields before reaching Prudhoe Bay, Alaska. We know the exact date, not to mention the hour, minute and second. We do not know its size. As with all tidal waves, it is safer to take precautions and move out to sea, where its arrival may not even be noticed. Disaster strikes those who are unprepared and caught near shore. There is little time left to mobilize, so to speak, and move the world's huge oil and gas fleet to the safety of the sea.

The authors

Scott M. Shemwell is director of Oil and Gas at MCI Systemhouse. He has more than 20 years of experience in executive management, information management and international business within the petroleum industry. He has written extensively on a variety of management subjects. Mr. Shemwell holds a BS degree in physics, an MBA degree, and a doctorate in business administration.

Jerry Dake is director of Systems Integration at MCI Systemhouse. He has more than 30 years of experience in operations management, information management and finance in the process industries, both internationally and domestically. Mr. Dake holds a BS degree in civil engineering, an MBA degree, and doctorate in industrial administration.

Bruce Friedman is manager of Systems Integration for the Process Industries Business Unit at MCI Systemhouse. He has more than 15 years of experience in client server computing and information technology management. Mr. Friedman has written several articles on lowering the total cost of technology ownership.

----------------------------------------------------------------------

-- Linkmeister (link@librarian.edu), November 02, 1999.

Linkdude and Engineer

There is also this Speech from Shell Oil John Mills. This was the real wake up call IMHO to the oil industry. While there is no mention of underwater problems (and there is) the scope is well defined. Somewhere there is an article that deals with the underwater problems, but the oil industry has been quite for a while and I forget the name. But one of the problems has to do with tides and bouency (sp?) which may be date dependent. The chips are located in the chambers that support the platform.

BEATING THE BUG THE INFRASTRUCTURAL CHALLENGES AHEAD

 John Mills

 Director of Corporate Affairs, Shell UK Limited

 The Petroleum Economist, 6th North Sea Conference

 21st April, 1998

  Beating the bug

A typical offshore platform or onshore gas plant uses 50-100 "embedded systems." These are sets of electronic code used to control equipment which are effectively sealed, and cannot be altered by the users. These systems contain anything up to 10,000 individual microchips. We have found that up to half of these systems are critical in terms of production and the impact of our activities on the environment.

-- Brian (imager@home.com), November 02, 1999.

Linkmeister and Brian:

Thanks for the links: I've read these before and what troubles me is the following. Both of these reports seem to have been written by "suits" rather then people who actually do the design and running of the rigs. I know from my own industry that there is a large knowledge gap between the front office and the field. Also in the link that Brian gave the writter said that the systems couldn't be accessed to be changed. Not that they couldn't be accessed to be replaced. Big difference there. Makes me wonder if someone down the line misunderstood something. Again, it really makes no sense to put something that is vital to an operation in a place that can't be reached if it fails. Failures occur all the time and if you make the system unreachable you've basicly set yourself up for failure. Y2K or no Y2K.

I may be hard headed about this but I would really have to hear it (and be shown it) from a platform designer or engineer who works on one before I buy into it.

-- The Engineer (The Engineer@tech.com), November 02, 1999.

"I can't stand it. Been reading the posts from today and I just cannot believe the way things are going. Even worse, a lot of it is financial as if we are all believing that there will necessarily BE an economy next year"

It's people like this who make me feel like a polly... perhaps I really am. Scary thing is - nobody here seems qualified to dispute his take on the embedded chip issue. And no polly here seems to be able to come up with any facts that disprove his opinion. Scary stuff. I've never understood how a date in a system that can NOT be changed could be significant. But then I have *NO* experience in embedded chips.. it's scary to hear someone who does, say otherwise. Obviously they know more about it then I do. IF he's correct, the potential here ... is unfathonable ... no grey area about it. I'd like permission to re-post the initial post from this thread in another forum - just another... is that ok?

Whitney

-- Whitney (y2kwhit@aol.com), November 03, 1999.

Whitney, Just another is out of town for awhile so will be unable to post a response to your request---HOWEVER permission granted by me (a scared wife doing the preps)

-- --just anothers wife (anotherswife@engineer.com), November 03, 1999.

Yikes. This looks worrisome.

-- cant (sleep@night.anymore), November 05, 1999.

Whitney --

Been away, but saw your post when I got home. (Wife pointed it out.)

Feel free to cross-post.

As to some of your concerns. The idea of what 'could' happen is, indeed, scary. In some ways, I may be the biggest 'doomer' on this, or any other site. When I get to considering long term implications of the potentials, you *do not* want to be in the same room with me. (Just ask my wife.) On the other hand, all you have to do to see some of the opposing viewpoints is to look at this thread. (Flint, and, for all that he works in a different field, 'The Engineer'.) There are others, who did not put in an appearance on this thread, notably, Cherri, who, again, appears to work in a different field, but often says some sensible things. Several others also have useful input.

The problem with 'refutation' is that all of us, me, Flint, The Engineer, all of us, are extrapolating *from what we know*. This colors our viewpoints. If nothing one ever worked on will have a problem, or if, like 'The Engineer', one works on physical applications in 'traditional' engineering fields, then the natural extension is that there will be no problem. If, like myself, virtually everything you ever worked on has the *potential* for problems, (believe me, I've spent more than a couple of sleepless nights trying to remember just what did get used as default cases,) then one will very naturally conclude that what everyone else did has the same potential. The problem is one of scope. There just *ISN'T* anyone with the perspective to see the whole problem. So we are *all* working from incomplete data.

-- just another (another@engineer.com), November 07, 1999.

First time reader,and contributor.Salutations to all the really good folk in this thread.Educational to anybody seeking information for better judgement.I come to this forum with a rare background:I designed,developed,manufactured,marketed and maintained a line of Programmable Logic Controllers in the mid-1970s with the Intel 8088 CP and EPROMS that had application on copymachines when a Xerox brand machine was 10 feet long,required special 220v wiring and cost a ton to operate.I came across one of my original machines at Region 3 GSA hq in DC two years ago.Still working.Absolutely will not survive the rollover.Can you imagine,a magnetic card access controller still functioning 25 years later? I even designed the two sided printed circuit boards and prototyped them myself.Brag,Brag.You can get my take on it on my website: http://www.braintherapy.comp/plc1.htm.

When I first came across the Y2K mess in summer 1998,I think it was a radio interview of Gary North by Art Bell,or with Ed Yourdon by Art Bell(heard both of those guys whenever Bell had them on),my reaction was mixed.After a suitable germination period,by late fall 98,I knew that I had better not fool around with this.

I had left technology completely in the mid-1980s with no interest in returning to it as a means to a living.My research into dyslexia and attention deficit disorder ADD with or without Hyperactivity,had led me into nutrition and its flip side,poison from lookalike edibles killing people left and right in America.

So for me,*preparation* involved more than just stocking up on cans of beef stew and lasagna,or Sam's 50# white rice and white bean bags.I knew that I would not knowingly ingest substances that I absolutely (no doubts)knew would destroy me,and or my wife.The purpose of survival preparedness is to survive alive.Right? I metasearched for organic foods to start my nutritional program in a box for myself and immediate family.The search brought buckets and bags without any thought of end-use packaging and/or nutritional systems-and even then,just a few sites.

I was forced by default to become the only source on the net with complete nutritional systems for meat(in this case, powdered whole organic eggs)eaters and vegetarians.In the meantime, drawing upon my sources developed during the dyslexia phase, I posted one report after another about toxic food elements. For example: the ubiquitos soybean attacks your thyroid. Will give you hypoglycemia (low temperature=confusion,apathy,low immune system,no sex drive,and on and on)and thyroid cancer.This from the FDA's (Food and Drug Administration)own Jefferson Arkansas toxicology laboratory.

I posted it as http://www.braintherapy.com/soya.htm.A lot of that kind of stuff is on my website.In the interests of preserving as many really smart people as possible in a worst case scenario,I strongly urge you to go to the soya report and get the inside story on it from the FDA(this time they didn't lie)and from Mary Enig Ph.D.,the world's leading authority on edible fats; and other equally off mainstream data.Nutrasweet, for example, is an immediate and totally absolute no-no if you value your neurons,dendrites,and synapses.Ditto monosodium glutamate. MSG is perfectly normal and harmless as it appears in nature firmly attached to a protein molecule. Strip off the protein and you get a neuroexcitotoxin.

Back to the embedded chip syndrome. The fundamental problem with embedded chips that will fail from Yr2K allergies, is that they defeat the MTBF (mean time between failure)and render all plans for executing MTTR (mean time to repair)obsolete. No matter your sector, power, rail, chemical, petro; Fix On Failure = developed plans with mean time between failure history and formulae. If you are tuned up for one failure per month or per thousand units of production, but instead you experience 2 such failures, you are shut down-kaput!Out of business. Why? Because you have only one engineer and one replacement embedded chip on hand. One of the failures remains failed for an indeterminate period until you have a FOF engineer and a spare component for the second failure. You might skate through on sheer luck the first time. But not the second, third, and fourth times. It is statistically impossible for key industries involved with PLC devices to turn on remote track switching servomotors, chemical process valves and vents, and the rest of whatever industrial process you mention, to survive repeated hits to their repair departments when the repair departments are staffed and inventoried pursuant to established MTBF experience. This is not arguable. It helps if you have ever had manufacturing experience as I have, and have had to write warrantee/guarantee policy for your customers' protection. Burton Linne braintherapy@braintherapy.com

I have sufficient supplies to feed me, my wife, single mom daughter and two grandchildren right on through until next summer while getting more and more food from my MicroFarm (new product on my website-easy to spot).

-- Burton Linne (braintherapy@braintherapy.com), November 27, 1999.

First time reader,and contributor.Salutations to all the really good folk in this thread.Educational to anybody seeking information for better judgement.I come to this forum with a rare background:I designed,developed,manufactured,marketed and maintained a line of Programmable Logic Controllers in the mid-1970s with the Intel 8088 CP and EPROMS that had application on copymachines when a Xerox brand machine was 10 feet long,required special 220v wiring and cost a ton to operate.I came across one of my original machines at Region 3 GSA hq in DC two years ago.Still working.Absolutely will not survive the rollover.Can you imagine,a magnetic card access controller still functioning 25 years later? I even designed the two sided printed circuit boards and prototyped them myself.Brag,Brag.You can get my take on it on my website: http://www.braintherapy.comp/plc1.htm.

When I first came across the Y2K mess in summer 1998,I think it was a radio interview of Gary North by Art Bell,or with Ed Yourdon by Art Bell(heard both of those guys whenever Bell had them on),my reaction was mixed.After a suitable germination period,by late fall 98,I knew that I had better not fool around with this.

I had left technology completely in the mid-1980s with no interest in returning to it as a means to a living.My research into dyslexia and attention deficit disorder ADD with or without Hyperactivity,had led me into nutrition and its flip side,poison from lookalike edibles killing people left and right in America.

So for me,*preparation* involved more than just stocking up on cans of beef stew and lasagna,or Sam's 50# white rice and white bean bags.I knew that I would not knowingly ingest substances that I absolutely (no doubts)knew would destroy me,and or my wife.The purpose of survival preparedness is to survive alive.Right? I metasearched for organic foods to start my nutritional program in a box for myself and immediate family.The search brought buckets and bags without any thought of end-use packaging and/or nutritional systems-and even then,just a few sites.

I was forced by default to become the only source on the net with complete nutritional systems for meat(in this case, powdered whole organic eggs)eaters and vegetarians.In the meantime, drawing upon my sources developed during the dyslexia phase, I posted one report after another about toxic food elements. For example: the ubiquitos soybean attacks your thyroid. Will give you hypoglycemia (low temperature=confusion,apathy,low immune system,no sex drive,and on and on)and thyroid cancer.This from the FDA's (Food and Drug Administration)own Jefferson Arkansas toxicology laboratory.

I posted it as http://www.braintherapy.com/soya.htm.A lot of that kind of stuff is on my website.In the interests of preserving as many really smart people as possible in a worst case scenario,I strongly urge you to go to the soya report and get the inside story on it from the FDA(this time they didn't lie)and from Mary Enig Ph.D.,the world's leading authority on edible fats; and other equally off mainstream data.Nutrasweet, for example, is an immediate and totally absolute no-no if you value your neurons,dendrites,and synapses.Ditto monosodium glutamate. MSG is perfectly normal and harmless as it appears in nature firmly attached to a protein molecule. Strip off the protein and you get a neuroexcitotoxin.

Back to the embedded chip syndrome. The fundamental problem with embedded chips that will fail from Yr2K allergies, is that they defeat the MTBF (mean time between failure)and render all plans for executing MTTR (mean time to repair)obsolete. No matter your sector, power, rail, chemical, petro; Fix On Failure = developed plans with mean time between failure history and formulae. If you are tuned up for one failure per month or per thousand units of production, but instead you experience 2 such failures, you are shut down-kaput!Out of business. Why? Because you have only one engineer and one replacement embedded chip on hand. One of the failures remains failed for an indeterminate period until you have a FOF engineer and a spare component for the second failure. You might skate through on sheer luck the first time. But not the second, third, and fourth times. It is statistically impossible for key industries involved with PLC devices to turn on remote track switching servomotors, chemical process valves and vents, and the rest of whatever industrial process you mention, to survive repeated hits to their repair departments when the repair departments are staffed and inventoried pursuant to established MTBF experience. This is not arguable. It helps if you have ever had manufacturing experience as I have, and have had to write warrantee/guarantee policy for your customers' protection. Burton Linne braintherapy@braintherapy.com

I have sufficient supplies to feed me, my wife, single mom daughter and two grandchildren right on through until next summer while getting more and more food from my MicroFarm (new product on my website-easy to spot).

-- Burton Linne (braintherapy@braintherapy.com), November 27, 1999.

Doncha just love these heavy-hitting "credentialed" doomish pronouncements as the fuse flames its fiery dash to the Finish Line?

-- Ashton & Leska in Cascadia (allaha@earthlink.net), November 28, 1999.

For Review purposes http://www.wbn.com/y2ktimebomb/Computech/Issues/mrtn9809.htm Why Is There So Much Risk In Year 2000 Embedded Systems Upgrades In Core Infrastructure Facilities? By Roleigh Martin March 2, 1998 The embedded system Year 2000 Problem with core infrastructures concerns equipment with embedded control devices used by utilities (power, gas, water, phone company equipment) that will operate improperly (causing equipment failure or worse) after the turn of the century, or in some cases after 1/1/1999 or even earlier dates. The 3/2/1998 issue of Business Week commented on this problem (underlines are mine): "In particular, electric utilities are only now becoming aware that programmable controllers--which have replaced mechanical relays in virtually all electricity-generating plants and control rooms--may behave badly or even freeze up when 2000 arrives. Many utilities are just getting a handle on the problem." The range of suspect equipment is vast--Figure 1 in my Jan/Feb 1998 article in the Year/2000 Journal depicts 25 examples of embedded systems to worry about. Most Y2K followers do not fully appreciate the complexity and enormity of the troubleshooting task ahead for these utilities. Two online documents are recommended, one from experts at TransAlta Utilities, Calgary, Alberta, and the GM Year 2000 Test Procedures. (The IEE has an equally good printed guide, but it's split apart among many html pages.) A piece of equipment may appear as a black box to an end-user, but contains upon inspection a multitude of "black boxes" (smaller embedded systems) from different vendors inside it. Furthermore, inside some of those smaller "black boxes," they may contain themselves a multitude of tinier "black boxes" from still different vendors and so on. Each "black box" can have up to ten layers of technology: Chips and microcode; pre-manufacture custom functionality; post-manufacture custom functionality; interfacing of devices; drivers; operating system; vendor-supplied application library; user-defined functionality; user integration of systems and devices; and business process associated with system use. From a design standpoint, to find out if a suspect device is Y2K compliant, many people have to be consulted for each "black box" component in order to cover these layers of technology. These consultants include microprocessor architecture specialists, firmware engineers, operating system engineers, device vendors, and end-users who oversaw the installation and/or modifications of the "black box" in the next larger "black box." The TransAlta document covers these points in depth and visually. Vendor statements are seldom available or reliable, either because of ignorance or falsehoods. In addition, the available engineering information is not always pertinent to Y2K compliance requirements or not available for all vintages (variations) of the equipment over the years (decades). Also, vendor statements can rarely address end-user implementation compliance issues. That is, a multitude of made-Y2K compliant devices might not be able to work in sync because they were made compliant in different, incompatible ways. Consequently end-user testing is required. See the GM Document, Figures 5.2.4 Year--2000 Component Test Report Form and Figure 5.2.7--Year 2000 Combined Component Test Report Form. The first has up to 48 tests to report on for individual component Y2K testing, also called by others "stand-alone component compliance testing." The second has up to 43 tests to report on for combined component Y2K testing, also called by others "implementation compliance testing." Yet, the GM document cautions: "This limited set of tests cannot prove a Component/System to be Year 2000 compliant, but using them will help identify several frequently observed problems. These test procedures are written as general instructions. Specific knowledge of the systems or components under test is required in addition to apply these test cases." Repeated testing is required to rule out fluke successes or fluke failures. Even in less rigorous Y2K embedded systems testing environments, the number of 30 tests per device is commonly recommended. Type testing, that is testing only one or a few of the same model number, has been documented among Y2K researchers as being sometimes unreliable because small variations can exist between one day's manufacturing run and another day's run. Testing is hampered in that dates cannot always be manipulated, some equipment cannot be taken out of production to be used for testing, and sometimes testing can cause as much damage as a postulated Y2K generated shutdown. In short, testing cannot always be done! In addition, the skillset needed to do first-class Y2K embedded systems troubleshooting is considerable and resources are scarce. What little equipment and tools are available are expensive. All of this can result in situations where the cost of testing is greater than the cost of replacing equipment. However even some new equipment is still not Y2K compliant and again, even if Y2K compliant, the multitude of new equipment might not work in sync with each other due to the various ways in which they were made Y2K compliant. So even if one replaces equipment, testing is still required. The bottom line -- there is a lot of risk and expense involved in ensuring that our core infrastructures will operate without failure after January, 2000.

-- ed portillo (magnacarta00@aol.com), November 29, 1999.

Continuing with a review of embedded systems http://www.wbn.com/y2ktimebomb/Computech/Issues/mrtn9809ii.htm One Year 2000 Embedded Systems Expert In Core Infrastructures Speaks Out Anonymously By Roleigh Martin March 3, 1998 This article continues where my article, "Why There Is So Much Risk In Year 2000 Embedded Systems Upgrades In Core Infrastructure Facilities," leaves off. Through my past experiences with the Y2K Problem, I have received some insider information. One embedded systems Year 2000 expert involved in the utility industry problem provided abundant insight. To protect his identity, let's call him "George." I know he is an expert from independent and public sources and I have verified it was he who sent me the material by phone calls I made to him. As for its authenticity, I can at least state the following: Either some in the industry are doing or saying the things said below, or else someone wants me to believe this information. Either way, it is worth pondering. George wrote about a November 1997 state utilities meeting with a senior executive of a leading engineering firm involved in the utility industry and a senior researcher on the embedded systems problem who is affiliated with an industry group. His observations follow--they have been "sanitized" by George to hide individual/group/corporate/geographical identities. He also included observations made subsequent to that meeting. "Those who have not yet recognized the embedded systems problems are already too late. It was said to take about 21 months to make one power generation plant Year 2000 compliant. 'The only thing we can be certain about the Year 2000 is that we won't be able to catch everything.' (Speaker with the State Public Power District who opened the meeting) The opinion was expressed that complete Year 2000 remediation for a number of utilities is an insurmountable task; therefore, they should just attempt to make the steps necessary to prove due diligence in the court of law." "It was apparent from the meeting that the majority of the electric utilities had little to no idea about the extent of their embedded systems problem. The $30-40 million figure to upgrade one typical power plant was given by the expert engineer who attended and spoke at the meeting. I have spoken with him about embedded systems on a few different occasions. The twenty-one month timeframe was first given by the engineering firm executive, and was concurred by the expert engineer." "The idea that complete remediation would not be possible was a strong theme in the expert engineer's presentation. In fact proving 'due diligence' was named as a major driver for participating in the proposed solution program. Due diligence was also named as a major driver in the engineering firm's executive presentation as to why the utilities should utilize the services of a professional engineering firm with expertise in doing embedded systems troubleshooting for Year 2000 upgrading of core infrastructures." "New HVAC machines (as well as others) do not always contain manual override for 'on' position. Work-arounds are impossible without costly replacement. Lead-time on replacement of such machines is already over one year and increasing." [An HVAC is a heating /ventilation/air-conditioning system. Without HVACs working, rooms containing electronic sensors and heat-sensitive data processing equipment can overheat and start malfunctioning.] "In a private phone conversation with the expert engineer, he expressed the opinion that a 30% outage is not out of the realm of reality. If thirty percent of the nation's grid goes, the rest may go as well. That is a reason that the few and far between plants who are ready (or at least closer to being ready) for the Year 2000 are considering isolation. That is not exactly an easy task. Also to consider: a customer who is serviced by a power transmission center rather than a power generation station could just be out of luck, even if his (far away) generation station is still pumping out power. This brings us to another point. Under current federal law, any power generator capable of producing excess power must be connected (whether on or off-line) to the power grid. This includes windmills (!) and private companies' power generation stations." "Every firm, of which I am aware, offering Year 2000 embedded systems remediation does not perform testing to a high enough degree... [One firm] tests only at the component level in embedded systems, and then only at the hardware/functioning software level. This leaves huge holes in their testing. They do not test for component interaction problems. There have been documented cases of interaction failures, often causing more severe problems than RTC or BIOS anomalies...yet [they do] not test for them. [They do] not review the ladder logic, which often contains date references, in PLCs (programmable logic controllers). They are also unaware of many problems in embedded systems which have been recently discovered. For instance, they were unaware of the cyclic rollover problem with certain PLCs... [One] must make sure the experts really are experts." "I do not plan to be anywhere near a big (or medium) city when the Year 2000 hits. There is just too much that could go wrong."

-- ed portillo (magnacarta00@aol.com), November 29, 1999.

One embedded systems Year 2000 expert = George = Mr CEO?

Hhhmmmm. Will it be Herstatt or Embedded or Both?

Just got the Chills. Love to all!!

-- justanthotherCEO (karlacalif@aol.com), November 29, 1999.

Of all the people who have contributed their time, talent, sweat, and mental energy to this incredible thread, Ed Portillo is the first to mention ladder logic (software analog to physical relay logic); a tutorial with schematics of "ladder logic" is on my website (reach it through http://www.braintherapy.com/plc1.htm).

Moreover, he is the first (to my knowledge from the two hours I have invested so far in studying this forum)to identify the fundamental killer to any polly attitude about embedded chips (of all kinds):they cannot be tested for allergies without taking them out of the system; and then cannot be relied on (notwithstanding whatever steps were taken to clear up the rollover allergy) until put back into the enterprise and retested for reliable operation, including especially, re-start after varying degrees of power failure. Without a scintilla of experience in power generation, I sense that it is impossible to test a power generator's immunity to Yr2K rollover allergens without stopping and starting offline. It must follow that it is not possible for any utility system or station system to perform an end-to-end test, blather to the contrary notwithstanding.

Portillo locked up his credibility with me in his closing remark above: "..make sure your expert is really an expert." However, he failed to take the next step: how do you qualify one as an "expert" in embedded chips, or any other engineering niche? From what I conclude from reading this thread, the contributors are mainly programmers, not hardware or firmware engineers experienced in starting with a pad of paper to wind up with a double-sided board with chips on it that turns a servomotor on and off in response to self-generated commands or remotely originated commands.

So that would be my initial question to a self-confessed "expert." ?What have you done with your own two hands and noodle to create something physical and electronic that will start and stop some kind of physical process upon instructions howsoever received? If you have, you still might not be an "expert," just an engineer doing his job within a limited context. If you haven't, then you cannot qualify as an embedded chip expert unless, (b) you have rebuilt or remediated a minimum 100 embedded chips systems with your own two hands, not as a supervisor/exec of a department; and additionally, can design, develop and prototype all four kinds of embedded chips if called upon to do so.

Full competence in Ladder Logic would be prerequisite to any consideration as an expert. This is so easy to master that I would dismiss as unqualified, anything coming from any "expert" without such familiarity.

Thirty minutes after reading Bruce Beach's bombshell initial report of his interviews with the Ph.D.s at Phillips Petroleum research center in Bartlesville Oklahoma, I concluded that continued residence on the seventh floor of a glass tower in Alexandria, Virginia inside the beltway about 5 miles from ground zero in the Pentagon's center garden would be tantamount to accepting willing victim status. On September 12, my lease on a small cottage with a well and septic system a mile out of city limits of an old town one mile wide x 2 miles long-population under 50,000, kicked in and my wife and I relocated our businesses and abode in a long one day marathon. We are now 45 miles from the Potomac River;15-20 miles from the mountains. An immediate benefit: the fire truck siren comes by once every 10 days on average, not 3 times a night.

I was a manufacturer in Chicago the day Martin Luther King Jr. was killed. I was there in the center of the city, not more than half a mile from the burning riots. A major core of the city's black community housing was totally destroyed. 35 years later, most of the destroyed commercial areas remain destroyed because no insurance company will underwrite liability, fire & casualty, and interruption of business policies. It took only 17 minutes into the power outage of 1977 in Manhattan for the looters to hit the streets. It took less than an hour more for the *law-abiding middle class* to hit the streets to get their fair share of the spoils. Now, integrate the foregoing and extrapolate. If you don't have a rehearsed escape plan from the city you live in, you may have made yourself a willing victim. Don't plan on using any of the usual routes-they'll be closed or jammed.

As for the Pollys, they are in a classic sucker's bet. They have everything to lose if they are wrong; nothing whatever to win if they are right. And the old Spanish proverb will operate on them with a vengeance: "And God said, 'Take what you want, but pay for it.'" if they are wrong and we get a worst case scenario, or anything close to it for at least 2 weeks. Thanks for your time reading this.Burton Linne

-- Burton Linne (braintherapy@braintherapy.com), November 29, 1999.