Myth - SEC reports indicate not enough of the Y2K budgets have been spent by the electric utility/telephone companies, therefore they are toast.

Fact - In my experience, it is the well run engineering projects that go under budget, typically because the technical problem is well understood and the scope growth is minimal. Both of these facts hold true for Y2K in electric/telephone utilities. Even the twists provided by the embedded systems aspect are turning out to be no big deal. This is because embedded systems (more correctly called micro-controllers) NEVER just freeze up due to the millennium rollover (yes, I know someone, somewhere, will probably find an exception or two, but that just proves the rule). In the area of budgets, the number and cost of remediation is significantly less that expected. You will soon start to hear that for some (many?) utilities, the Y2K problem would not have caused a loss of power to a single customer at all, even if left unremediated. So of course many are underbudget.

Prediction - The NRC will not shut down any nuclear plants due to Y2K.

While it is true that the NRC does have the authority to order the shutdown of a nuclear plant, that authority is not absolute. Proper due process must always be given, and Y2K has not shown to be a threat to nuclear safety that even comes close to being a situation where an ordered shutdown would be warranted. And please, remember that NIRS is an "watchdog" organization that would like to see nuclear power go away for any reason, so any statement from them about Y2K and nuclear safety is suspect (Note that the reverse is not true for nuclear power workers. While we would like to see nuclear power continue as a significant portion of this nations energies source, nuclear plant workers who falsify safety related documents could be subject to personal criminal penalties.).

bob

-- Anonymous, March 29, 1999

Answers

bob,

You seem to be holding some information that would greatly relieve fears concerning Y2K and the electric industry. Would you point me to a report or some more details on how the embedded systems testing went. I would like to see the total number of problems found and the total of those where were deemed to be mission critical. I would also like to know the total number of test cases that have been sucessfully completed and their general scope. Would you also share with us the total number of embedded systems on the grids(an estimate would be fine). I feel much more confident with quantified assessments. I would also like to know the number of utilities that have completed their Y2K remediation and have returned to business a usual. If you have this information, please share it with me. You wouldn't have to take a negative aproach to communicating this information, but without it, I have little to go on. I want to make up my own mind as you have made up yours.

-- Anonymous, March 30, 1999

Bob, if what you say is correct then at least some utilities will be releasing some of their y2k staff to work for utilities that started late. I have yet to hear of one such case. If what you say is true, then why are there so many utilities still in the remediation phase?

Please release some of the data behind your conclusions. Thanks.

-- Anonymous, March 30, 1999

Reporter,

Nothing like making it clear what you would like. Let's see what is out there.

Yes, as a utility insider, I have access to Y2K information that is not public yet, but which will be coming out soon. This information is not only for my electric utility, but many others that I interface with. Remember, past Y2K information, even from my own utility has unfortunately been filled with "useless words". This is starting to change for a number of reasons. First, we have a very good understanding now of exactly where we stand with respect to Y2K. Second, the industry results are surprisingly similar, sometimes with whole utilities reporting that NOTHING would have caused a loss of power to any customer due to Y2K, even if left unremediated. Third, and last, is the growing realization that as far as Y2K and embedded systems are concerned, the public fear over Y2K might cause more problems that the Y2K effect on the infrastructure itself.

Where can you verify some of this information right now? Here are two ways.

First, go to the organization that has the most influence over any electric utility. Not the SEC, but your states PUC (Public Utility Commission). Even with deregulation coming up, they still call the shots, and the utilities answer. We have already received our list of Y2K questions. Because they are inherently localized, they ask the specific questions you also want answered for your area. And because they answer to the public, the hearings and results are open to the public. There is probably a mechanism in which you can ask questions yourself, and you will be asking the Y2K experts, not the customer service reps or receiving the form response you normally get.

Second, talking specifically about embedded systems, let's take a look at the facts? Rick Cowles explores possibilities, but to my knowledge he has never identified a single actual problem device. Gary North is just hyping fear by stating if we don't know the all the answers yet then Y2K must be able to kill you. Again, not a single actual problem device is identified. What do you personally know about Y2K? Did any of your household devices freeze up or your computer crash. No. I assume we have all gotten past the point of believing that your car will stop, your microwave will malfunction or an elevator will halt between floors. Now lets expand to other embedded systems. Yes, plenty of industrial devices have cosmetic Y2K problems, but not a single one just freezes up due to rolling over to 1/1/2000. Besides me saying this is what all of the Y2K workers are seeing in actual testing, here is a way you can maybe help yourself believe this.

Gartner Group says there are over 25 billion embedded systems in the world. Let's call it 250 million that fail completely. Where are they? Look at vendor Y2K status on their web site. Many are pretty informative, and today, very accurate (compared to our internal testing results). Your see statements like "fails leap year progression in power down mode", "can't manually set the date past XX/XX/XX", or "will not display year 2000 and above", but you never see a statement that "this devices halts dead in its tracks". Logically, it is very difficult to prove something doesn't exist, but if the embedded systems Y2K problem is as widespread and severe as some would have you believe, how come no one is finding these types of failures?

It was possible that the effects of Y2K could have been very severe. The effects of Y2K on software are well documented and could still be a major problem. Everyone still needs to look at their embedded systems. However, the number and severity of embedded system problems just don't exist, but no one is going to take on the task of trying to prove this negative in one nice package. Sorry.

bob

-- Anonymous, March 31, 1999

Bob, you wrote, "Yes, plenty of industrial devices have cosmetic Y2K problems, but not a single one just freezes up due to rolling over to 1/1/2000." Unless you don't believe the CIO of General Motors, then that is not true. Last year there was a Fortune Featured article titled, "Industry Wakes Up To The Year 2000 Menace". I'm not sure if it's still online but I copied it to a Word document. It was a very long piece, but here are some paragraphs from that article:

"Many computers go into the equivalent of catatonic shock when they read a year date of 00: The machines and devices they drive stop, print out wrong information, or malfunction in other ways. Averting a millennium mess entails more, however, than correcting year 00s to the full, four-digit 2000 wherever they are found. Innumerable interconnections between software programs must also be checked to ensure that a program inscribed with 00 does not, in the manner of a computer virus that plays havoc with your PC, corrupt a program that has been fixed."

"Unfounded gloom and doom? Not if you listen to Ralph J. Szygenda, chief information officer at General Motors, whose staff is now feverishly correcting what he calls "catastrophic problems" in every GM plant. In March the automaker disclosed that it expects to spend $400 million to $550 million to fix year 2000 problems in factories as well as engineering labs and offices. Or consider the words of Rob Baxter, Honeywell's vice president in charge of making his company's line of industrial control products "year 2000 compliant," to use computer industry jargon. From what he has seen among Honeywell customers, Baxter fears that "some plants will have trouble operating and will have to shut down. Some will run at a reduced scope. I expect considerable system outages during December 1999 through February 2000."

"Small wonder, then, that many plant managers and their bosses plan to stay close to their jobs over the three-day weekend when the millennium rolls in. Already they've had a foretaste of what could go wrong. A somewhat similar time problem--programmers' failure to account for the 1996 leap year--halted some production lines at the beginning of 1997, causing millions of dollars in damage. In simulations of the transition from 1999 to 2000, some factory robots, as well as computers that control electric power generation and transmission, stopped dead. "

"There used to be very little off-the-shelf software for shop-floor use, Swanton explains, referring to types of programs such as manufacturing-execution, material-handling, and inventory-management systems. As a result, he says, "a lot of systems integrators and company programmers wrote applications themselves. Lord only knows what's buried inside those VAX minicomputers and other machines."

"So for a long time manufacturing companies snoozed, including GM. When he arrived at the automotive giant a year and a half ago to take over the CIO job, recalls Ralph Szygenda, he was amazed "that most people assumed that the factory floor didn't have year 2000 problems." Szygenda, with experience in manufacturing at Texas Instruments, didn't settle for assumptions. He shook GM out of its slumber by turning to outside companies such as Deloitte & Touche and Raytheon Engineers & Constructors, specialists in solving the problem, which sent in 91 experts to assess the automaker's situation. Supplemented by squads of GM technicians and programmers, these experts fanned out through GM's 117 facilities in 35 countries. What they found shocked even the factory-wise Szygenda.

"At each one of our factories there are catastrophic problems," says the blunt-talking executive. "Amazingly enough, machines on the factory floor are far more sensitive to incorrect dates than we ever anticipated. When we tested robotic devices for transition into the year 2000, for example, they just froze and stopped operating."

"The troubleshooters cannot relax because they know what's at stake. A host of developments have offered chilling previews of what laggard companies can expect when the year 2000 rolls in:

 Leap-year snafus damaged production lines when programmers failed to account for the extra day in February 1996. At a small U.S. manufacturer of industrial solutions that prefers to remain unnamed, production ground to a halt on Jan. 1, 1997. Before workers could remedy the situation, the liquids hardened in the pipelines, which had to be replaced at a cost of $1 million. That caused late deliveries and the loss of three customers. A similar leap-year oversight caused $1 million of damage at Comalco's aluminum refinery in Tasmania, when controls at all smelting-pot lines shut down, damaging five pot cells beyond repair."

" Dry runs are revealing a threat of cutoffs in vital services. When the Hawaiian Electric utility in Honolulu ran tests on its computer systems last year to determine if the year 2000 bug would have any effect, the systems quit. This meant that some customers could have lost their power or received it at a higher frequency, making clocks run faster and damaging or destroying valuable production equipment in factories. Communications systems could crash too. The U.S. Army's Materiel Command, testing its PBX telephone systems for the 2000 transition, found that they ran fine for three days after the turn of the year, until accumulated date errors shut down the whole network. "

"GM's Szygenda recently told his company's suppliers: "No matter where you are in the journey, one thing will not change--the deadline. Dec. 31, 1999, will come regardless whether we're prepared or not. I'll bet there are quite a few CEOs out there that will be sweating as the big ball drops in Times Square."

If the CIO of General Motors witnessed "catastrophic problems" and the Honeywell VP has grave concerns about the rollover to 2000, then I'm not going to bet they were somehow exaggerating. And if you really think that all businesses are investigating any Year 2000 problems they might have, or that those who are will all finish in time, then you should know about this nice bridge in Brooklyn that's for sale...

-- Anonymous, March 31, 1999

Bonnie, An excellent response as usual. I have been reading your posts for some time, mostly lurking. You have one of the best handles on the problem I have seen on the internet. I read the fortune magazine article, and in fact also have it. Excellent. Personally, I'm betting on Chrysler over GM to make it. Several years ago they did a y2k test at one of their plants and it shut down crash. In addition, it locked everyone into the plant and they couldn't get out. They learned. If you noticed they just announced they were going to make and market an electric bicycle which goes up to 20 miles and 15 mph. Could this be our transportation of the future?

-- Anonymous, April 03, 1999

Bonnie,

Here are a few thoughts, and lets define terms first.

Let's call an embedded system (more properly called a microcontroller) something that has a microprocessor in it, but might not even look like it has a microprocessor in it, and is typically smaller than a standard PC box. Let's use the term Distributed Control System (DCS), to describe something that might provides high level process control to an electric generating plant or a manufacturing plant. Typically, hundreds, thousands or even ten thousand embedded systems might connect to one or a handful of DCSs. These DCSs look and smell like a computer. They have operating systems, video screens and applications.

My point is not that DCSs never lock up and shut down the process (i.e. trip the generating plant or stop the production line). I am sure they can (although it is not very common in the electric utility environment). My point is that, so far, no one has found even one of the 100-10000 embedded systems that connect to a DCS just freeze up.

Why is this important? It tremendously reduces the scope of fixing the Y2K problem. DCS Y2K problems are almost always fixed by a software upgrade, not a major system replacement. The minor Y2K problems with hardware in the field can almost always be left as is if need to (almost everyone is fixing almost everything though, for a number of reasons). This is why I continue to believe that the potential infrastructure problems associated with Y2K have been completely overblown.

You are definitely seeing this viewpoint shift in the electric industry. What started as a health report that nation wide blackouts will almost surely affect the nation, is getting more and more positive over time. You will never see 100% assurance that no one will have a temporary loss of power. But you never had that assurance in the past, either.

This is also not to say that you should totally disregard personal preparedness. But in my opinion, if you are fleeing the cities, buying a years supply of freeze dried food, buying gas generators and gas storage tanks just for the Y2K problem, you are going overboard, especially if these preparations are taking away from your children's college funds, your retirement planning, or any other situation that you will be facing in the future.

bob

-- Anonymous, April 03, 1999

Bonnie:
 
The entire article which was published by Fortune Magazine is still online.
 
http://w ww.pathfinder.com/fortune/1998/980427/imt.html
 
Dan

-- Anonymous, April 06, 1999