I talked with a friend who is y2k honcho at a large automobile assembly plant. They just finished their last test, after remediating over 7000 systems (this is manufacturing equipment, not back office systems. One plant). It took 70 people 18 months to do this, and required replacement of PLCs, computers, software, and operating systems. He tells me that the interrelationships were nasty, and that without the assistance of a supergeek engineer, they'd have missed a lot.

Interestingly, he reports that normal testing would not have uncovered many of the problems they found. The symptoms would have been strange behaviors and breakdowns for some period of time, not obviously y2k related by either symptom or time exhibited.

I wonder where GM stands?

-- Flint (flintc@mindspring.com), February 08, 1999

Answers

Like most news these days, this is very good and very bad. Good that the plant has finished. Very, very good, in fact.

Very bad that normal testing would not have uncovered many of the problems. What sort of extraordinary testing was done, Flint?

Is there enough industry 'togetherness' that the core of this team can be used to lend expertise to others -- at least within the same company (I assume that they have more than one assembly plant.....maybe not)? At the very least the supergeek enbgineer should get out a "lessons learned" paper.

-- De (dealton@concentric.net), February 08, 1999.

Hmm.......I give new meaning to the phrase "couldn't spell engineer, but now I are one."

-- De (dealton@concentric.net), February 08, 1999.

What a GREAT story !! Any possibility of sharing with us the name of this facility. I'm sure, based on their success, they would be more than happy to share this info with us and their stock holders.

Ray

-- Ray (ray@totacc.com), February 08, 1999.

I am sure that there are a lot of competent people out there working on this. The problem is that there are also a lot of incompetent people out there working on this. Not pejoratively speaking, just saying that as you have related, were it not for this 'super geek' the problems may easily have gone undetected. As you have said, in this case, they were 'lucky' to have this super geek there that found problems that mortal programmers would have missed. There are not enough of these super geeks to go around.

You have said that 'normal testing' would not have uncovered these problems. It was the result of a recent survey that over 70% of all entities would NOT be doing proper testing and many no testing at all.

Testing is ABSOLUTELY essential and most will not do it properly and on top of that we find that the tresting that is exemplified here, indicates that without the presence of this 'ubergeek' they would have had many undetected problems.

The upshot of your geekvine indicator is very very bad news. In one sense it is beyond me that geeks have not already factored in the signifigance of their actual inability to get the job done in time. But, on the other hand one must also factor in their typical over- estimation of their abilities.

Where does GM stand? I'll tell you straight out. They have not a snowball's chance in hell of making it. It is not physically possible. The more complex a system, the greater is its vulnerability. I can not imagine a more complex set of interrelationships than those that would be had by about the world's largest company. This is not even to mention the web of external dependencies that are VITAL to their operation. This ALONE would take GM out of business.

There is NO WAY that GM is going to overcome thirty years of software engineering results. They are not going to be that entity which disproves tens of thousands of hours of IT Metrics. And enough of their vendors, VITAL to GM's very existence, will not disprove those FACTS either.

At some point, it will become MATHEMATICALLY obvious, even to an overly optimistic geek, that the job will not be done in time, that he is on a death march and that to remain is dangerous for him and his family. The 'Great Geek Migration' will commence sooner or later.

It is only a matter of time.

-- Paul Milne (fedinfo@halifax.com), February 08, 1999.

GM has 100,000 suppliers. Remember the recent strike? It only takes one or two missing parts and they cannot make a car. They have robotics and embedded lines of code (loc) to go through. Based on IT history and software metrics, for every 600 lines of code changed there will be one new error introduced inadvertently. That is about 170,000 new errors introduced for every 100 million loc altered. There are ten months left. The latest estimate I saw said they were budgeting $550 million, but that was last summer - they may have raised it by now - which everyone does as they realize the size and slope of the mountain they are climbing.

They are working hard, but the facts above argue that they are not going to make it. One number that would help us ascertain status is how much they have actually spent versus what they have budgeted - assuming that their lawyers would be telling the truth to the SEC in the 10Q filing - what did I just say - lawyers and truth in the same sentence? I think I better get some non-decaf before I go near any live data today.

-- Rob Michaels (sonofdust@net.com), February 08, 1999.

That third sentence got screwed up. It should have said that they have robotic and embedded systems to go through, and Two Billion loc. Off to get that coffee now. Rob

-- Rob Michaels (sonofdust@net.com), February 08, 1999.

Here's another salient and recent example:

Last week, the power plant at Ford's Rouge facility blew up. Personnel casualties aside, the plant was idled for a few days before temporary power was brought in. The effects of the Rouge facility being down for a few days rippled through many of Ford's other facilities, and caused line shutdowns due to lack of parts.

Figure it out, people. We're a "just in time" society. The economy can weather an isolated instance such as the Rouge facility problems. When you've got hundreds or thousands or hundreds of thousands of these problems happening at the same time, you've got major economic issues rippling through the entire economy. Interesting times, indeed.

Thanks for the original post, Flint.

-- Dan Webster (cant@spamme.com), February 08, 1999.

Damn. This pushes me up one at least notch on the Diane scale, maybe two based on the implied troubles in petrochemical and refinery industries. (Can you spell plastics? Raw material - like nylon? Lube oil?) Food processing too? Don't know - last time they reported, only 3 of 500 companies in food industry gave the feds an answer.

I was really hoping that embedded chips were a minor, easy to find to problem. Not counting on it, but really hoping. Also, that testing embedded chips and process-level controller would be simple. Apparently that isn't the case.

Good news on the remediation effort. Please pass along my appreciation for his efforts. One person truly can make a difference.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 08, 1999.

" One person truly can make a difference. "

Think of what Einstein and a few others accomplished. Instead of building on their ideas for the good of man, in a responsible manner, mediocre and greedy minds misused them.

Is there a parallel in the IT?

-- fly .:. (.@...), February 08, 1999.

I'm sorry I can't tell you much more than I wrote.

I can add that company lawyers prohibit public release of any details pertaining to any specific noncompliant equipment, for fear of lawsuits by the equipment vendors. Confidential information is of course being shared both with the vendors and with other automobile manufacturers using the same equipment (which is all pretty common). Replacements for most of it are readily available, but by no means cheap!

Though my friend didn't specifically say what problems would not have shown up simply by turning dates ahead, I can make an educated guess based on my own experience in manufacturing. JIT operations require a huge amount of data collection. If, say, your defect-rate tracking is hosed and you don't know it, you can find yourself up to your ass in alligators a few months down the road, bombarded by inventory errors internally, and dealer complaints externally. And this is only one of zillions of such little systems that add up to a workable operation.

And oh yes, based on their experience, this company is not about to announce compliance anytime soon, if at all. They plan to continue with the "we don't expect any problems" line indefinitely. Safer that way.

My friend has no clue about the status of their business systems. This is only the equipment side.

-- Flint (flintc@mindspring.com), February 08, 1999.

Well Flint I guess this means another trip to Sam's Club! Also do you have a bullet proof vest handy?(remember the other post)Thank-you for the information because this is what is needed to make the hard decisions. Take care Flint. Tman...

-- Tman (Tman@IBAgeek.com), February 08, 1999.