The lack of obvious embedded system failures (how and why)

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Like most of you I have been watching the rollovers throughout the world with mixed feelings. I am happy that nothing bad has happened. After all its not every day that you see your worst nightmare avoided. By that I mean watching power go out as the clock strikes midnight around the world.

However, like most of you I feel a little mystified at the almost complete lack of embedded system failures. Most of us have felt that _something_ should happen.

I've been dredging through my memory for some type of explanation. Here it goes.

The lack of failures today was the combination of a number of factors. Each of these things played a part, but were not by themselves alone responsible for the lack of problems today.

1) The work did pay off. Many people have been working very hard on the problem. They were not able to fix all of the problems but they were able to fix many of the big obvious infastructure critical ones.

2) Electric utilities were never seriously at risk. This just confirms many of the things Dick Mills (and others) said in his earlier articles. There were not many y2k issues in electric utilities and the issues that existed could be worked around. Those that could not be worked around were fixed. See 1) for more on this.

3) Embedded system failures do not always cause immediately fatal events. Many embedded systems simply log dates. Incorrect date logging does not cause a system to shut down or blow up. However, it can cause data corruption and make a company unable to track things. Not nearly as obvious but just as problematic as an explosion. See the Wired Magazine Texaco remediation article. Many embedded systems may have failed today, their results are simply unseen.

4) Many companies have shut down their systems for y2k. I can't find the link for this but seem to remember reading articles of this kind. Gordan Gecko's posts on oil pipelines shutting down come to mind. These may fail as they are brought back up. (I belive this was documented in the Embedded Systems Fault Casebook but I can't check it as the site is currently down.

5) Many more companies are actively monitering the rollover. Problems are contained before they become visible to the public. They are able to implement backup analog systems or shut things down before they get too screwed up. This shows that the worldwide warnings about the y2k bugs were heeded and necessary steps were generally taken.

6) Things are pretty quiet today. No real strain has been put on the systems. Businesses have not worked in 2000 yet. Critical infastructure can and does work without problems under todays low load and constant supervision. Problems may crop up under a normal day's strain.

7) We were lucky. I feel better about what will follow after today. I can handle an economic disruption, at least, I can as soon as I use my cash-on-hand to pay off my y2k debts.

8) Infastructure critical business failures were inherently rare. None of us had any solid numbers for embedded system failures. Nobody even knew how many there were! Our failure to see the lack of threat does not reflect poorly on any of us. Too many variables to make an accurate prediction.

Just some thoughts. Flame away.

-- John Ainsworth (ainsje00@wfu.edu), December 31, 1999

Answers

John,

Thanks for categorizing things in a nice, rational fashion. I think you've covered most, if not all, of the bases.

Ed

-- Ed Yourdon (ed@yourdon.com), December 31, 1999.


Thank you for at least trying to offer some explanation.

I've been watching this (from my bunker) and I couldn't figure it out.

It looked . . . supernatural!

-- Ariana Christopher (pergados@yahoo.com), December 31, 1999.


John,

Logical,concise, based on information. I don't understand why everyone thought the lights would go out. Especially after reading the death by a thousand cuts and the fact that most plants would shut down at rollover.

I am laughing out loud at the people that have thrown in the towel and even now listening to the .gov beating their chest.

Wow. Dumb, dumb, dumb, impatient,impatient,impatient

-- d...... (dciinc@aol.com), December 31, 1999.


I buy answer number 4. I also truly believe that no one, not the media, not the gov't or corporate America will admit to an embedded chip failure unless something catastrophic happens that cannot be covered up (although they do a good job with that too). Don't expect an absolution here.

-- Mello1 (Mello1@ix.netcom.com), December 31, 1999.

John,

Thank you for your sober and measured posting. I have been banging my head against the wall since 11:00 am GMT wondering why so few failures in embeddeds. I mean, a failure rate that is so small that it is almost negligible. However this does not square with the bench tests ...

Midnight was only one of the measuring posts, and I feel that we have passed a major hurdle with flying colours. However, every ounce of logic is saying 'it aint over till the fat lady sings', and methinks she will clear her throat on Tuesday ...

-- merville (merville@globalnet.co.uk), December 31, 1999.



Aren't you all a bit premature 3 hours before midnight here? And media totally busy with fireworks elsewhere? Some processes take awhile to run away. W

-- W (me@home.now), December 31, 1999.

The embedded systems programmers always, always said there was no problem. Check the deja archives for newsgroup comp.arch.embedded They have maintained for years that there would not be a problem

Meanwhile you bozos were giving credence to Bruce Beach. Yow!

    --bks



-- Bradley K. Sherman (bks@netcom.com), December 31, 1999.


Arianna: Do you like to mudwrestle?

-- King of Spain (madrid@aol.cum), December 31, 1999.

What the hell is that oaf BKS doing here.

BKS f**K off back to csy2k will ya.

Most of us are here because we can't stand the sight of you.

-- Andy (2000EOD@prodigy.net), December 31, 1999.


I think Andy is saying happy new year the only way he knows how. So same to you Andy. And many many more!

-- Flint (flintc@mindspring.com), December 31, 1999.


In my opinion, we all can easily see the computer problem, purely mechanical in nature involving various aspects of a simple coding error. However, I am convinced there is a much deeper underlying spiritual problem with the human race which is hidden from general view. I am convinced there is a God who made us and to whom we each in his own way are accountable. Men everywhere have forgotten God. We are about to be reminded. Noah was locked in the Ark seven days before it began to rain and flood. That was a real long week. I imagine his family laughed at him and called him nuts. Sincerely ..Tom

-- Tom Format (tom format @ aol.com), December 31, 1999.

We shutdown everything early this morning at work. The computer geeks start at midnight to bring everythin up. All systems are to be tested over the weekend BEFORE the official workday starts. Everything could work or some things could be toast. Major industrial loads were removed from the system for the rollover. This provides an extra cushion for the utilities and makes things look quite. I would be curious how the oil restarts go before I look to unload my preps.

Also it is common practice to focus attention on those things that have a high probability of succeeding (ie electricity) to provide cover for anything that doesn't.

Happy New Year one and All.

By the by do we get the best athletes of the 20 century again next December?

-- Squid (ItsDark@down.here), December 31, 1999.


And a very happy new year to you and yours Flint.

Bradley can f**K off.

-- Andy (2000EOD@prodigy.net), December 31, 1999.


John, Thanks. Tom, Very well put. I didn't know that Noah had to wait.

-- Mara (MaraWayne@aol.com), December 31, 1999.

C-SPAN 9:30 pm Eastern:

General Willard - Joint Chiefs of staff stated "some Y2K glitches have occured overseas. None of them have interupted the flow of supply and all remain in the 'Green status'".

On follow-up General Willard stated "overseas governments are implementing contingency plans and applying work arounds. The reason for no interuptions is most likely that 'they have their first team working on the job'".

So questions as I see it are:

1. How long does it take to complete a workaround and restore a system?

2. As bad data builds or systems are running on manual, might a divergent trend deteriorate control?

3. What might happen when the 'first team' needs much needed rest?

It is still too early to say Y2K remediation globally is a complete success, although the absence of apparrent immediate interuptions excellent reason for optimism. Peace to all.

-- Bill P (porterwn@one.net), December 31, 1999.



Heh, Andy, keep running...

-- Hoffmeister (hoff_meister@my-deja.com), December 31, 1999.

Finally an intelligent post shows up!

What can we determine right now at this point but that the grids are remaining up? That is only one piece of this mega-puzzle, but a key one. We are all so very happy to see the lights are remaining on time zone after time zone--but that's it? Show's over?

I don't think so. Re-read points #5 and #6 above. I don't know a thing about embeddeds, and that's ok, because if not one of them faults out we still have some other pieces of the puzzle to deal with. Data corruption doesn't necessarily explode in an instant, like 00:00:01 on 01/01/00. It accumulates.

-- Remaining Vigilant (but...@gonnaparty.tonight), December 31, 1999.


Heh, Andy, keep running...

-- Hoffmeister (hoff_meister@my-deja.com), December 31, 1999. ======================================================================

What do you mean by that Hoffy?

BKS chooses this day to turn up on this board with his antics?

Leave it out pal.

Come back in 3 months Hoffy, we expected better of you.

I guess you believe all the AP?UPI?Reuters stories that it's all been fixed.

What a dazzling mindset you and BKS have Hoffy.

-- Andy (2000EOD@prodigy.net), December 31, 1999.


Thanks for the kind words everyone. Sorry I took so long to get back, I got headed off by a party.

Happy New Year!!

-- John Ainsworth (ainsje00@wfu.edu), January 01, 2000.


Andy??

You seem a bit miffed ol' chap.

Problems??

I wouldn't let it bother me too much. You guys are batting 1.000 with your predictions.

That alone is something to be proud of.

I stand in awe......

Deano

-- Deano (deano@luvthebeach.com), January 01, 2000.


Deano - use your brain.

has your "institution", fanny mae or whatever it is , worked in the real world yet.

Done any "month- ends"???

Then shut the f-up.

So let's see we have y2k "I've figured it all out in 10 minutes" pro AND

"I've figured it all out in 30 minutes" Flint...

Morons the pair of them.

Stick around Flint/Deano, 1 trillion dollars wasn't spent to fix a ZX 80...

-- Andy (2000EOD@prodigy.net), January 01, 2000.


Andy,

God bless you son. You really are ignorant and I thought it was all just an act.

Month end runs??? Of course we have silly!! We've completed over 75 monthend runs tonight alone. Hell, they even spanned the rollover and balanced to the penny. We're talking a few billion dollars here so that's a lot of pennies.

Year end runs?? Of course we have silly!! We've completed around 50 of those at this point in time. Oh yeah, they balanced too.

So. I won't shut the fuck up. I plan on riding your sorry ass for quite some time you see. I put up with your shit and name calling for months. Guess what??? It's my turn now Andy.

And Andy?? It ain't FNMA.

Can we chat more later???

Deano

-- Deano (deano@luvthebeach.com), January 01, 2000.


Deano.

And you are in charge of a y2k project???

-- Andy (2000EOD@prodigy.net), January 01, 2000.


Beano,

I find your attitude incredible.

Hawaii hasn't even celebrated rollover yet and you, Mr. PROJECT MGR. at fanny mae, or freddi frump whoever the hell gubbmint give away our taxes agency you work for...

Start crowing about how successful you've been...

Incredible...

Has your agency interfaced with any others yet? Taxes? Banking payments? Receipts? Month end? Rollover status?

You are an AOK asshole buddy.

It's Saturday night and you know precisely

DICK

about your agency and how it will cope in the months ahead.

I would be insulting morons to put you in their category.

Your brains have been fried on the beach my clueless one.

-- Andy (2000EOD@prodigy.net), January 01, 2000.


Anyone ever bet on a long-shot at the track?

It's amazing; people REALLY DO that! 100-1 even! From my few times there, EVERY horse gets SOME money bet on it. Pure idiocy as any reasonable person could tell. That horse ain't gonna come close! But they do it anyway -- just can' figure it.

Don't you think they should build a pen at the track for the people who bet on the losing horses, and give the winning bettors and other spectators some rotten tomatoes to gather around and throw at them?

Then finally we'll get those tote boards RIGHT, with big ZEROES on most of the horses and all the money on the CORRECT few.

-- jor-el (jor-el@krypton.uni), January 01, 2000.


Andy

You are, without a doubt, the most ignorant individual I have ever come across in my 40 years. You (obviously) don't have a clue, and then when someone trys like hell to give you one, you don't listen.

That, I'm afraid, translates into a "lost cause".

I tried to explain to you the other night that we had performed monthend and yearend runs. We had (already) transmitted data to ALL OVER THE WORLD by that time. That data was processed with NO PROBLEMS. Do you GET IT yet son???

You and your fear-mongering buddies were wrong. But I'm sure you'll keep your fingers crossed.

And, for the umpteenth time for our 'special friends' - it's not a government agency. It's a publicy traded corporation.

Get help Andy before you hurt yourself, or worse, someone else.

Deano

-- Deano (deano@luvthebeach.com), January 03, 2000.


Moderation questions? read the FAQ