However, anyone in the industry who understands the total scope of the problem, and who will speak off the record, is scared to death.

Rick, I quote you above, will anyone speak off the record on your open forum (or is that a contradiction in terms). I find the electrical issue to be the most secretive of all y2k.

Many Thanks Richard

-- Anonymous, October 28, 1998

Answers

I am an electronic engineer working for a controls company who supplies the control systems for power generation prime movers. The company I work for is a leader in this industry, doing over 500M in sales per year.

I would like to address the Y2K issue as related to our products. To do this, let me start off by giving a brief discussion of what our products do in the power generation application. The base for providing power is a prime mover (engine or turbine). The prime mover uses fuel to produce rotational energy. This rotational energy is translated into electrical energy through a generator. To vary the power produced by the generator, the amount of fuel to the prime mover is varied.

My company's basic products control the fuel to the prime mover. Many of these devices are analog or mechanical in nature, and are not affected by the Y2K rollover. I would estimate that 40-50% of our generator set application sales are these type of products.

Stepping up a level, are the base digital controls used to control the prime mover power. These digital controls have no real time clock, and operate directly from the core processor operating system. These products, therefore, are not affected by the Y2K rollover either. These products account for about 40% of our generator set application sales.

On the top level are the complex digital systems. These systems may have real time clocks, and may interface with Unix or OS-2 operating systems, and therefore may be susceptible to the Y2K rollover problem. However, the use of date coding in the applications software is not integral to the operation of the equipment. There is no function in the software which links machine operation to a date. Worst case would be an incorrect date display on a operator control panel screen.These complex systems are typically used on critical applications (large power producers, satellite tracking systems, etc), and have manual backups which allow continued operation of the prime mover in the event of a total control system failure. The sales of these high end digital systems make up the final 10 -20% of our generator application sales.

In many applications, especially the lower end products, there are typically some sort of upper level controllers that take care of starting/stopping, and monitoring the prime mover. I can make no statements regarding the health of these products at the time of the Y2K rollover. However, even if these systems were to fail, manual operation of the prime mover would still be possible. In many systems, this type of failure mode is accounted for, and the system essentially has auto/manual switch configurations for this purpose. Switching to manual operation totally bypasses any upper level control commands, and puts operation of the prime mover directly in the hands of the operator . In systems where this type of failure is not accounted for, the generator could be brought back on line by bypassing any inputs provided by the upper level controller with manual command inputs provided by jumper wires, etc to the base prime mover control. It is a rather sloppy way of taking care of the problem, but it does work (believe it or not these type of system failures occur even without the Y2K problem!).

I doubt that we will experience any major power outages in the US due to the Y2K bug due to failures of the prime mover control systems. This might all be a moot point, however, if those Russian computers controlling the warheads have a Y2K countdown problem....

Jeff

-- Anonymous, October 28, 1998

Jeff, thanks for the response. I'm still at a loss though as to why the power companies seem to be so secretive about the issue. Fear of panic perhaps? My own power company sent a friend a letter that they had a plan in place to take care of their y2k problems back on July 22nd. On that same day responded to a PUC commission survey saying that they had't finished assessing the problem. Maybe that's why they're so quiet?

I'm curious to know if the problem lies not with the source of the power generation happens (as you've explained shouldn't be a problem) but rather getting that "power to the people." So to speak. (?)

BTW there's an article regarding the state of Russia's readiness in the news section at this web site on October 21st. http://www.coolpages.net/2000/

thanks

Jeff

-- Anonymous, October 28, 1998

Jeff,

I appreciated your very well thought out response to this question. My question to you is, however, why won't the utilities just explain the problem the way you have and let us all relax. If there is a very small chance of any problems, as you have layed out so well in your explanation, why doesn't the government or the PUC people just issue statement like yours? Why do they keep dodging the issue and, as Sen Bennett has done, quietly warn us to prepare?

Linda

-- Anonymous, October 28, 1998

Interesting that you stated the systems could be run manually. When was the last time they were run manually? Will you be able to prove that you can run these systems manually? If we can run these systems manually then wouldn't it be prudent to do it for lets say for a week next year to prove to the American public that we have nothing to fear and that power will be available. And I would have to reiterate the previous statemants on why the utility companies are being so tight lipped on their progress? Do these systems rely on the telephone lines to operate?

-- Anonymous, October 29, 1998

I've heard this same explanation myself from the IEEE. I agree I don't think the plants will fail at the core level. Most are as you say well engineered and are not "directly" affected by computer malfunctions. As to running them manually? Or if need be running many of them manually I don't know if most of these plants even have the staffing to do that anymore? Are the crews that run these plants trained to run them this way? In my opinion the real threat to the plants comes from without. The huge vendor chains these "industrial" devices need to operate are at risk. We know that many med. and small businesses haven't even begun to do their y2k repairs. These machines need constant maintainence and repair to run 24 hrs. a day 365 days a year year in and year out. Even then I agree most will be up and running in jan 2000.We can look at Russia today as an example of a society in a state of collapse at many levels but the power is still on! So I'm optimistic but worried. I still plan to have some kind of back up system for my home in case of black outs or brown outs. It's going to be winter here and even a day without power could destroy my home if it's really cold. The power companies are really trying to dodge the bullet on this issue and it's only making things worse. What we need is just the kind of straight forward explanation as you have given. I think it coupled with each companies on going compliance statements and schedules could go a long way to calming rising fear of a grid failure.

-- Anonymous, October 29, 1998

The electric transmission and distribution systems do utilize embedded systems many different ways. But there are also far more devices installed that are electronic or rely on magnetics to function. These devices are not susceptible to Y2K problems.

Control systems that I have designed and installed all have a method of manual control and a means to disable the embedded system (failure happens under normal operation from time to time)if necessary. The electricians at my company are trained in manual operation of the system, and actually do so routinely.

My group has inventoried and assessed 279 embedded systems installed for control of the transmission system we operate. All have been certified Y2K compliant by the manufacturer. In addition, integrated system testing is being conducted on facilities to ensure proper operation into the year 2000.

It's important for me to point out the interconnection of the electric system and our dependence on telecommunications. Even though one utility may be confident of their position, it's difficult to communicate that because external factors can have a such a large impact.

-- Anonymous, October 29, 1998

Jeff, a belated thanks for your explanation, this is what I had hoped and secretly thought. Now to convince the survivalists, we can concentrate on the issues that really are affected by y2k. Do the utility companies share information on y2k, eg post findings on the web, its an issue which is really outside of normal competitive considerations. William & Jeff these are the few real postings I've seen from the technicians who lets face it are the only ones who know. Why don't utilities just provide an authorative explanation such as yours to put everyone's mind at rest.

-- Anonymous, October 30, 1998

But then you hear of:

"When asked about the availability of their vendors to supply parts, he admitted that vendor supply is short. Even more alarming was the testimony of Terry Harman from Alliant Energy[western Wisconsin, northern Illinois, eastern Iowa and southeastern Minnesota], who is far ahead of NSP. [Note that Alliant is smaller than NSP] They have already identified 170,000 embedded chips that need to be replaced, which he believes comprise 98% of all the chips that need to be replaced, and when asked about vendor supplies he responded, "some of our suppliers have told us to go away." He stated this was a VERY serious issue. "

So what is the real story??? ^o-o^

-- Anonymous, October 30, 1998

I have asked Nick where he read the information in his quote via e-mail, but found this quote in a "news clipping" dated 10/9/98 on the State of Wisconsin web-site.

"There's always the potential for something really unexpected happening," said Bob Newell, project manager of year 2000 planning for Alliant Energy, parent company of the former Wisconsin Power & Light Co. in Madison.

A team of more than 200 employees and consultants has checked most of the 170,000 devices in the utility's system and the failure rate so far has been only 1 to 2 percent, Newell said. He expects Alliant Energy to be ready by the end of January."

Quite a different picture - Y2K is truly a conundrum. It is difficult to prepare when it seems no-one will know exactly what to prepare for until the rollover.

-- Anonymous, October 30, 1998

Jeff Chellieff:

Thank you for your insightful comments, particularly since you seem to have the expertise and background to render an opinion on the subject matter. In the vein of Ronnie Regan's famous "Trust but verify" motto, I would like to hear your (and anyone else's, particularly those with expertise in the matter) comments on the following points.

There is a very informative article by Mark Frautschi, Ph.D. titled "Embedded Systems and the Year 2000 Problem (The OTHER Year 2000 Problem)," which can be found at http://www.tmn.com/~frautsch/y2k2.html. Dr. Frautschi makes the point that even embedded systems which do not need to keep track of the date may be nontheless noncompliant because OEM's buy off-the-shelf chips with greater capacities than needed (i.e., RTC) for economic reasons. Here are some excerpts:

"The problem exists in a surprisingly large number of systems, particularly with systems with no design requirement to keep a date. Why is this? The reasons are largely economic. It is very expensive to engineer a custom integrated circuit. These non-recoverable engineering costs can exceed $100,000 including the salary of an ASIC (Application Specific Integrated Circuit) designer and several engineering runs at a commercial chip foundry. When manufacturers of embedded systems need to incorporate any form of real timing capability (seconds, minutes, etc. - as opposed to system clock cycles) into a system, they face a "build or buy" decision. In the case of time sensitive chips, they will generally purchase an off-the-shelf, general purpose, timing chip, (or the rights to its design). This costs about one dollar."

"The same economics drive the manufacturers of these timing chips to develop a "one chip fits all" solution for their customers, the Original Equipment Manufacturers, or OEMs. In this case the OEMs utilize a general-purpose chip that is more versatile than they require. After accessing only those capabilities that they actually need, they embed these chips in their products. There is no gain in reinventing this very common wheel. This results in capability significantly beyond the OEMs' design requirements being embedded, including date keeping, as indicated below."

***

"The logic "It does not need to keep dates, therefore it does not keep dates," has no basis in the internal operation of the chip. This has resulted in a number of systems being declared Year-2000-compliant when in fact their firmware has not been tested. The question is not "Does it need a date"?, the question is "Does it use time in any way?" Examples of systems containing unassessed code include remote control load management switches installed at consumer sites by electric utilities, automobile power train transmission control modules and major household appliances."

1. I understand your explanation that your company's products control the fuel to the prime movers. I am curious, however, as to (A) whether the digital controls use any factor of time to regulate the flow of fuel, and (B) whether your company uses "off-the-shelf" "one-size fits all" chips, and if so, has anyone bothered to check whether the chips themselves are compliant notwithstanding the fact that the application has no design requirement to keep a date?

2. You stated that "the use of date coding in the applications software is not integral to the operation of the equipment. There is no function in the software which links machine operation to a date." It is my understanding (layperson's) from researching this whole y2k business as it affects the power industry that power companies adhere to a strict, exhaustive maintenance schedule which is time sensitive. It is my undertstaning that there are fail-safe systems built in so that if no maintenance is performed on a piece of equipment on time, that equipment is automatically, and without human intervention, shut down until the problem is addressed. Thus it would seem to me that there has to be a function in the software which links machine operation to a date.

Your thoughts??

-- Anonymous, October 30, 1998

The question was not directed to me specifically, but I would offer the following:

1. If the device does not have a battery, what will the time be after power down and back up again? Even if there is battery backup, all embedded systems that I've worked on can be set to whatever time the operator chooses. If someone felt their embedded system was suspect to the Y2K problem, they likely would "back date" their equipment until a suitable work-around was found.

My car is controlled by an embedded system; who sets the time and how is it accomplished? 2. I do not know of any utility that owns devices which shut down because someone forgot to do maintenance on it. There might be devices out there that alert, but I've never seen one that would shut down.

My car doesn't shut down if I forget to change the oil.

-- Anonymous, October 30, 1998

Jeff's(Chellieff@msn.com) response--informative and valuable-- leads me to wonder about two elements in it. First, he endson a note about Russians controlling warheads. I wonder if he would be specific as to which of the following scenarios seem likely to him and why: missiles are lauanched; second scenario: computers indicate a launch but it is inaccurate; third: computers simply go down. I ask this because reports I have heard seem to be most concerned about scenario number two. Jeff do you see another one as more porbable and why? Second, assuming power grid is okay due to manual control and backups, etc.; what would you think (if embedded chips don't bring it down) could bring it down? Simply put, assume it goes down (brown outs, blackouts. What, Jeff, seems likely to you to have caused it?

-- Anonymous, October 31, 1998

Here's some off the record for you:

...These paragraphs are from a utility insider who attended a state- wide session on Y2K and embedded systems in electric utilities in a Mid-West state in 1997. I have confirmed most of what this individual relayed to me via email correspondence with others who also attended that session. He reported that a leading electric utility engineering firm official and a leading Y2K expert with a leading electric utility industry group told the utilities the following:

In the testing of two coal-fire power plants (which were currently offline and being used as "hot spares") for year 2000 compliance, the clocks were simultaneously rolled over to the year 2000, causing immediate plant failure. In an attempt to better understand the failure, the roll over test was repeated. In the second test, the plants again failed, but a different embedded controller was determined to be at fault. The roll over test was repeated a third time in hopes of replicating one of the previous failures. In this test, the plants failed from yet a different embedded controller. It was determined that this last failure would have caused a grid-wide failure had the plants been online. It took 13 days in order to restore the plants to working condition from the last failure.

OFF THE RECORD:

The discussions that took place in the meetings really scared the hell out of me. It seems that the power utilities that have not yet recognized the embedded systems problems are already too late. It was said to take about 21 months and $30-40 mil to make one generation plant compliant. The utility companies at this meeting expressed the opinion that compliance would not be possible due to budgetary and time constraints. "The only thing we can be certain about the year 2000 is that we won't be able to fix everything." The opinion was expressed that complete Y2K remediation is an insurmountable task, therefore utilities should just attempt to make the steps necessary to prove due diligence in the court of law.

According to this official, the money that all utilities report they are spending for Y2K is outrageously low. The problem is that the current tax accounting laws encourage the utilities to only publicize their software Y2K budget --because those costs have to be expensed out in that year, not depreciated over many years. Thus many utilities who are trying to be diligent are reportedly lumping the bulk of their Y2K equipment replacement and/or upgrade expenditures into normal equipment upgrade costs. Consequently, this data is hidden amongst non-Y2K costs and neither the public nor the regulators can distinguish if they are doing a first class job or a really cheapskate, risky one. I've talked with both Y2K tax attorneys and managers at utility companies and I know this is happening, to my great frustration.

I have also been told by managers working inside these utilities that they do not want their shareholders to panic over the full amount of money they are spending on Y2K upgrades. Personally, I would think educated shareholders would be more willing to hold onto their utility stock during the Y2K crisis if they knew the utilities were spending more on the problem, but apparently the utilities feel otherwise.

Consequently, it is impossible for an outside Y2K expert to grade these utilities on the quality of their Y2K work simply by reading their SEC reports and by hearing their Y2K presentations. Most of us studying the Y2K utility problem are totally in the dark and that includes the regulators. How frustrating! I do know that some utilities are reportedly taking a FOF approach (Fix On Failure) and that really concerns me due to the 13 day downtime incident illustrated above. (Yes, I've had a few Y2K utility project insiders write me and tell me the management is engaged in type testing or plan to include use of the FOF approach. This might actually be acceptable for non-mission critical equipment, but one Y2K project manager felt too much was being excluded from testing at his company.)...

by Roleigh Martin:

-- Anonymous, November 03, 1998