Question for Ed.

Have you managed to unearth any material for the completion of this chapter. I haven't seen much hard evidence that there is an embedded systems y2k threat to the electricity or water supply. What is the general opinion of electrical engineers working in the industry after all its the only one that counts. Many Thanks Richard

-- Richard Dale (rdale@figroup.co.uk), October 16, 1998

Answers

Richard, what you and many others seem to not understand is where we should logically put "the burden of proof", so to speak. The automated processes that are used to provide electricity and clean water are riddled with both embedded and non-embedded systems. It may be that there does not exist "hard evidence" to your satisfaction that there is a Y2K threat, but considering what is at stake, we should be demanding hard evidence that Y2K is NOT a threat! It is absolutely irresponsible to take any other position, in my opinion.

-- Jack (jsprat@eld.net), October 16, 1998.

I would say innocent until proved guilty. Rick Cowles reckons there could be "problems we don't know about" but they haven't found any! Anyway the only opinion I would respect is that of an expert in the field without being hamstrung by the "official view". Where are they, challenge to the audience. y2k is a subject IMHO that demands experts to make the judgements, I know about the particular systems I have worked on (business software), I don't know about power generation, neither do I suspect do the majority of posters to this forum. The opinion of those people on the subject is not even remotely valid.

-- Richard Dale (rdale@figroup.co.uk), October 16, 1998.

In a court of law, we require a high burden of proof on the prosecution because we recognize that if a mistake is made, it will mean that someone will be wrongfully incarcerated or put to death. A Y2K problem with our life-sustaining systems could result in much misery and many deaths, if not outright collapse of our civilization. Doesn't common sense say that we should instead require that these systems be proved innocent beyond a reasonable doubt? (Or, at a bare minimum, that contingency plans be ready, that have been proved innocent beyond a reasonable doubt?)

-- Jack (jsprat@eld.net), October 16, 1998.

Jack, to me that's just plain common sense. But I don't have credentials to prove that I have a common sense ;)

-- Chris (catsy@pond.com), October 16, 1998.

Oh, except that I'm an RN, and my common sense comes in handy when you're one of my patient ;)

-- Chris (catsy@pond.com), October 16, 1998.

I agree w/ Richard on this one. The only opinion that counts is that of the experts. The Gartner Group's Senate testimony last week stated that electric power problems in the U.S. will be "isolated and minor." They also said that "embedded systems will have a limited effect on Year 2000 problems, and we will see a minimal number of failures from these devices." They did not disclose where they got the info. to come to that conclusion because of agreements they made with the info. providers. So, unless we trust the Gartner Group as speaking for the experts, we really haven't heard from the experts.

I want to hear from the experts.

-- Buddy Y. (DC) (buddy@bellatlantic.net), October 16, 1998.

Why are the National Guards of Wisconsin and Iowa being put on alert. A lawyer for a power company in Wisconsin said people should buy generators. I have not talked to one power company that said they would guarantee power. Embedded chips were the problem. Why does Ed have to prove anything? Just make a few phone calls and you will get an education in a hurry. Take the commons sense approach. I don't have proof that an earthquake will hit here tomorrow, but it could!

-- Dave (dave22@concentric.net), October 16, 1998.

Because Ed has written Timebomb 2000 with a non-existent chapter on the effect on utilities. If there is a real problem why don't we know about it. The book has to be based on fact. Lets hear from the engineers, good news or bad, have they been gagged, don't they know?!

-- Richard Dale (rdale@figroup.co.uk), October 16, 1998.

A valuable source of information, obviously, is "the experts". But to absolutely hold that the public at large cannot make an informed decision, leaving it to "the experts" to make these decisions for us, shows that we are in a sad state of affairs indeed. Especially when: 1) "the experts" do not agree among themselves; 2) "the experts" may have their own agendas, to wit, trying to wiggle out of a huge mess that is coming that they should have known about and foreseen. I want the DOG to wag the TAIL: I want the PUBLIC to demand that "THE EXPERTS" examine our life-sustaining support systems and determine what effect, if any, Y2K will have. And, further, that the evidence and conclusions be open to review and third party verification.

-- Jack (jsprat@eld.net), October 16, 1998.

I think we're really agreed in principle, I don't however assume a situation unless its based on fact. Seems like the facts are missing. I don't want politicians or pundits to come to conclusions not based on fact. Whether or not the "experts" provide the facts i don't care, but can't see who else can. I wouldn't trust an official compliance statement for instance entirely, but if I had inside information.... That inside information has got to come from the guys working on it on a daily basis. Or even the people who built/customised the things.

-- Richard Dale (rdale@figroup.co.uk), October 16, 1998.

Well, now I think we're getting somewhere. I agree with the last statement that you just made Jack.

Point of clarification: The "experts" being referred to here are the engineers who are actually running the power systems and the designers of systems used in running the power systems. I couldn't care less about the opinions of anyone else, whether they be historians, economists, or, in this case, IT professionals. I guess Dick Mills at Westergaard could be considered one, Rick Cowles another, but we need more. We need to hear from the people actually running the plants.

-- Buddy Y. (buddy@bellatlantic.net), October 16, 1998.

Richard: I'm not sure how things work in the UK, but here in the US, virtually every employer big or small requires that an employee sign some form of a 'non-disclosure agreement' at the time they are hired which expressly forbids that employee from disclosing any 'unapproved' information about the company.

Thus, unless an IS employee gets 'approval' -- which is highly unlikely if the information could potentially hurt the employer -- or posts anonymously using one of the many sources available on the Internet ('whistle blowers'), we are unlikely to hear directly from such individuals. Still, some whistle blowers will continue to come forward.

But as any lawyer in this country will quickly confirm, employers are very, very serious about 'non-disclosure'. (It is one of the key foundations of competition in the marketplace. Competitive advantage can be fleeting. It can make billions for your company or put you out of business quickly if you don't have it.)

Consider also that, in this country at least, we have have a very strong tradition of crucifying 'whistle blowers'. Protections are superficial at best. Extreme courage, combined with a total disregard for your own income, career, lawsuits, and general well being would be required. There's just not enough people who are that brave/foolish to get the job done.

Relying on whistle blowers to provide ALL of our direct data is not sufficient.

This is another reason why I support truly independent, 3rd party auditing and verification for those business (i.e. utilities, banking) on which our lives depend. Such auditors must have direct access to all employees in an organization and the freedom to interview them without restrictions. To make this work, immunity from liability/prosecution will need to be granted both to the businesses and the employees.

I really don't see many other options here. I agree that 'proof of innocence' must be the standard to which our life-sustaining businesses are held. Again, I'd like to point out that I'm not talking about ALL businesses here. I couldn't care less about Joe's Widgit Corp unless my life depends upon Joe's widgits. I do, however, depend on my power company's widgits.

Arnie

-- Arnie Rimmer (arnie_rimmer@usa.net), October 16, 1998.

Well, Richard, exactly what situation are you ASSUMING, then? That power and clean water will be just fine? Based on what?? That they have been fine as long as you have lived, and so far no one has come up with "hard evidence" that Y2K will change that??? Even if, say, the probability that Y2K will cause problems is low (personally, I believe it to be high), surely everyone will agree that the RISKS are unacceptable. Which, to me, translates to: ASSUME that Y2K will cause problems, until proven otherwise .

-- Jack (jsprat@eld.net), October 16, 1998.

Jack on what basis do you believe y2k risk to be high! Have you got any evidence?! I have no opinion on the subject I just want the facts somehow. Once I know the facts I will form an opinion. Do YOU know the y2k effect on a particular Insurance company I worked with in the UK, of course not. I know the facts. I would not presume to know about the y2k readiness of your company, I expect you do.

-- Richard Dale (rdale@figroup.co.uk), October 16, 1998.

It is the lack of verified information that is most disturbing. Y2K is most certainly an issue of relative risk.

Let's make it a bit more personal with this imagined scenario: My wife has just been seriously injured and needs immediate medical attention. There are two roads leading to the hospital. One is 10 miles long and, 9 miles down the road, has a bridge over a deep canyon. The other is 20 miles long, straight and flat all the way. At the Y in the road, two people are arguing. Both just happen to be structural engineers with years of experience. One says that she has just examined the bridge on the short path and that it is extremely dangerous and could collapse at any moment. The other says no, he has also examined the bridge and while it's a bit shakey but in no danger of immediate collapse. Do I take the long way or the short way?

By the way, how many here have seen the famous film of the Tacoma Narrows bridge collapsing? We learned a lot from that one. Harmonic resonance is not unique to bridges.

Arnie

-- Arnie Rimmer (arnie_rimmer@usa.net), October 16, 1998.

Here is a good summary of the Embedded chip problem:

REFERENCE 1: The King Kong of Y2K, by Jim Lord October 12, 1998 http://www.y2ktimebomb.com/Tip/Lord/lord9841.htm

The true nature of the Y2K story has been misunderstood. After all the effort put into chasing down those pesky little monkeys (mainframes and desktop computers), a Nine Hundred Pound Gorilla has emerged on the scene. He is called embedded processor and he looks like one very tough dude. ..... These are some of the more common embedded processor uses and potential failures. There are as many more as there are creative programmers in the world. As the results of testing become known, it is increasingly clear that the embedded system component of Y2K is, by far, the most important part of the problem. One recent study, for example, indicates that the embedded processor component of Y2K could be as much as two to four times as severe as the mainframe and desktop components COMBINED.

REFERENCE 2: And here are some articles on the Power Grid, including how it is likely to handle y2k type of disruptions:

Another Myth, SCADA & EMS Failures Would Crash the Grid - Part 2 By Dick Mills October 16, 1998 http://www.y2ktimebomb.com/PP/RC/dm9841.htm If you enjoy learning how everyday things really work, you're going to enjoy today's column. If you like pictures, you'll love it. We are about to finish last week's column about SCADA (Supervisory Control and Data Acquisition) and EMS (Energy Management Systems). The context, is the critical functions of grid operations and how difficult it may or may not be do get along without them if need be.

REFERENCE 3: Dick Mills' Y2K Power Prognosis http://www.y2ktimebomb.com/PP/RC/dmpp.htm

I believe that the electric utility can fulfill its primary mission in 2000. I define the mission as, "Provide power to most of the people most of the time in 2000 so as to avoid disastrous injury, suffering, business failures or unemployment due to lack of electricity."

In other words, we can avoid causing the end of civilization by blackouts, but the year 2000 will hardly be business as usual.

REFERENCE 4: Power Failures in 2000, by Dick Mills http://www.y2ktimebomb.com/PP/RC/rc9825.htm

There are two distinct kinds of threats to the supply of power in 2000; blackouts and power shortages.

A blackout occurs when an unexpected event has consequences that cascade into one event after another until catastrophic collapse. Falling dominos is a good visual metaphor. The most famous such event was the great northeast blackout of November 9, 1965. The twin events in 1996 that disrupted power in 12 western states and provinces are a more recent example.

Blackouts can be local, or regional, or continental in scope. Noone knows how many Y2K blackouts to expect, or how widespread they might be. However, there are some things we do know, without resorting to crystal balls.

No matter how big the blackout, we can restore service to just about everyone within 24 to 72 hours. The operators can operate some misbehaving things manually and make do without others.

Some authors have predicted one month or longer blackouts because of Y2K. Poppycock! The power system protection systems are aimed at causing things to trip at most incongruities as a precaution so that equipment is protected from damage and power can be restored rapidly. This makes blackout events more frequent, but shorter in duration.

-- Jon Miles (jon@milesresearch.com), October 16, 1998.

Well, if you're like most people, you stand there waiting until the ambulance arrives and let them make the decision. If you can make your own decisions, you decide which route affords you the most control over the outcome |)

-- Damian Solorzano (oggy1@webtv.net), October 16, 1998.

Richard, only since you asked, the reasons that I personally believe that the risk of major (read meltdown) Y2K problems for our life- sustaining systems is high is based on my perception of what I have learned since becoming "Y2K convinced" some months ago. Understand, then, that what follows is JUST my personal view, and actually is completely independent on my earlier posts on this thread, which just took the position that there was merely SUFFICIENT evidence that there MIGHT be Y2K problems, and that in view of the tremendous UNACCEPTABLE RISK, that is what justifies a "Y2K guility until proven innocent" assumption. == OK, I think that the chances are real high because: 1) Common sense tells me that since many utilities have supposedly been hard at work trying to fix Y2K for years, but no one is there yet, even at this very late date -- its a real problem, a very pervasive one, and one that no one is denying (anymore); 2) The electric utilities, by virtue of the so-called Power Grid, are very interconnected and interdependent in ways that even "the experts" have never, until now, had to worry about, and (as the NERC report notes) loss of a few could bring down all of them -- my personal take is the high complexity and low understanding makes for high failures; 3) Senator Bennet said it best at the conclusion of his hearings last June, that if Year 2000 were to come at that time, we would have no power -- other than NERC's "cautious optimism", and the recent Gartner Group's recent claims based on so-called "freestanding" processors, a very peculiar term that I have not seen before, I do not see any real improvement here; 4) My take is that "the experts" are real good at their own area, but very ignorant on others, and so when an electric utility expert says that they can go manual if necessary, s/he is assuming that the necessary telecommunications will be there to achieve this synchronization with other plants; 5) Other miscellaneous dependencies that an electric utility "expert" cannot take into account, such as transportation of coal, assuarnce that nuclear plants will be permitted to continue to operate (they won't if telecommunications is down), that the banking system will be there to pay people, etc., etc. ==Sorry this was so long, Richard, but you asked... ==

-- Jack (jsprat@eld.net), October 16, 1998.

I'd feel alot more confident about power in 2k if not for the fact that our power company is currently (8-) warning us that we should expect blackouts this winter. They are asking us not to run our Christmas lights and to avoid the use of electricity in the 'prime hours; of 4 to 8 pm. The problem is that Alberta's economy has grown faster than the power generation ability, so if we have an economic slump, the short term problem may be solved. One of our dailies ran an article yesterday that said (in part) that "the electricity surplus has fallen from 15% to as low as 1% ... over the past few years". At any rate, if they can't maintain power this winter, things really don't look good for y2k. And it's COLD outside!

-- Tricia the Canuck (jayles@telusplanet.net), October 16, 1998.

Jon: Regarding your statement:

"No matter how big the blackout, we can restore service to just about everyone within 24 to 72 hours. The operators can operate some misbehaving things manually and make do without others."

I would like to believe this was true but I suspect many of the people who survived up 2-week outages during last years ice storm in the northeast would probably beg to differ.

This is not say that that some things can't be done maually or that some things can't be ignored altogether. But remember that a great many other significant events will be occurring simultaneously.

-- Arnie Rimmer (arnie_rimmer@usa.net), October 16, 1998.

1. Arnie: Just to clarify, the quotes in references 2, 3, & 4 were from Dick Mills, not me. This is the bio for Dick Mills :

Dick Mills has been creating software for power plants and power systems for more than 30 years. He was a pioneer in operator training simulators, helping five different companies get established. He has more than 2000 hours of simulator control room experience in emergency and startup conditions. He created the first independent network simulator, PTI's PSS; still the industry standard 25 years later. Dick also designed turbine controls, integrated plant controls, the automatic generation controls used in energy management systems, nuclear power plant probabilistic risk assessment tools, and process monitoring, archiving and optimization systems.

Dick clicked to the Y2K problem in 1997. He switched careers to work full time on Y2K in power grids. He is presently a consultant at Compaq's Year 2000 Expertise Center, although he speaks here as a power engineer, not a Compaq representative.

Mr. Mills can be reached by email at dmills@albany.net or on the web at http://www.albany.net/~dmills

2. Since the system is designed for failover, it seems reasonable to expect at least some power to be available, however spotty and insufficient it may be. The bigger issue is the availability and delivery of other forms of energy such as coal, gas (all kinds), nuclear sources, etc. which rely on trucks, trains, and ships. This refers to both the domestic and imported sources of energy.

3. The bow-wave impact of the y2k problem (which is already noticeable) is economic (the fractional reserve system and the mass- panic factors), and the wake effect will be impaired manufacturing, production, and distribution. For more info on this see these two articles:

http://www.pei-intl.com/TOPICS/LEVERAGE.HTM

http://www.csis.org/html/y2kw4.html

-- Jon Miles (jon@milesresearch.com), October 16, 1998.

Arnie said:

"I would like to believe this was true but I suspect many of the people who survived up 2-week outages during last years ice storm in the northeast would probably beg to differ. "

Keep in mind that the ice storm in the northeast took down power lines and poles, etc. This will not be the case for Y2K.

-- Buddy Y. (DC) (buddy@bellatlantic.net), October 16, 1998.

Jon: Thanks for the clarification. Sorry, my mistake. Still, I would like to believe Dick Mills is correct and it does give one reason to be 'cautiously encouraged'.

But I think about the analogy given in another thread here about if 10,000 pit bulls were chasing you and 100 of them suddenly fell over dead, would you stop running? No, but I would be most grateful if the 100 that did keel over were the biggest and meanest of the lot. ;)

There's an old bluegrass tune called "The Preacher and the Bear". The story has a bear chasing a preacher who'd gone out hunting and found something a bit bigger than he'd bargained for. Part of the chorus has the preacher lamenting "Now Lord, if ya can't help me, for goodness sake don't you help that bear."

Buddy: You said

"Keep in mind that the ice storm in the northeast took down power lines and poles, etc. This will not be the case for Y2K."

Well, I agree that this is the case for 'short outages' but in large cities and/or prolonged outages, refer to "How many BTUs in a utility pole?" and other similar posts. What this means is that the longer the outage actually last, the more collatoral damage our critical infrastructure will incur -- whether it be from short-sighted/frightened people, riots, unintentional fires, etc. That collatoral damage will only compound the time required to restore service.

But let's accept Mills 72 hour figure for the sake of discussion. What would be the effects on a city the size of Los Angeles? The answer of course is "It depends." It depends on whether the city was basically prepared for such an outage. A 72 hour outage is difficult but, if prepared for, also easily survivable with minimal loss of life. It need not be catastrophic. If even half of the people there prepared, social chaos would be minimized. But if only say 5% prepare, even a 72 hour outage could be catastrophic and Milne may well get his "7-11/toast" scenario. All things considered, some preparation would be far better than none at all. Preparing for a 72 hour outage is cheap, easy and will not unnecessarily frighten people. It's the due diligence thing. Arnie

-- Arnie Rimmer (arnie_rimmer@usa.net), October 16, 1998.

Can anybody refresh our collective memories? What happened in the big blackouts in history? Were there widespread riots, etc.? The ice storm last year didn't result in any, did it? The New York area was blacked out for quite a while, what happened then?

-- Buddy Y. (buddy@bellatlantic.net), October 16, 1998.

What no one seems to be considering here is that the power grids were created for ecomomic reasons, NOT technical ones.

It would seem reasonable to assume then, that decisions regarding the design and construction of the various grids were made on the basis of "HOW MUCH MONEY WILL IT MAKE ME?", rather than those questions which an engineer might ask regarding the character of the physical grid.

Still, in a capitalistic economy, the profit motive is priority number one and profit goes away if the grid fails.

Does this perspective clarify the picture or muddy it?

-- Hardliner (searcher@internet.com), October 16, 1998.

Arnie,

No utility poles in NYC... :-)

-- anon (anon@anon.com), October 16, 1998.

That is an interesting possible insight. I certainly have no idea, of course, whether the grid came about due to economic rather than technical considerations, but based on every other experience I have ever had, I could well imagine the "techies" begging and pleading that the interconnectedness would introduce a lot of risk that would otherwise not be there... == Anyone from the electric industry out there to give us an education here?

-- Jack (jsprat@eld.net), October 16, 1998.

"Anyone from the electric industry out there to give us an education here? "

I think that's what Richard's original question was...

-- Mike (gartner@execpc.com), October 16, 1998.

I'll ask the original question again, there must be some electrical engineers on the internet, surely you can reply using someone else's EM address if you wish to remain anonymous. Correct me if I'm wrong but none of the posters so far are qualified. If I had wanted the general opinion of the world at large I would have asked for it. I also did not ask for irrelevant speculation. I'm not saying that you are not entitled to your speculation and opinion, its just that on this occasion I did not request it.

-- Richard Dale (rdale@figroup.co.uk), October 19, 1998.

Buddy,

<< The New York area was blacked out for quite a while, what happened then? >>

The birthrate went through the roof nine months later.

-- Paul Neuhardt (neuhardt@ultranet.com), October 19, 1998.

The national "grids" came about through happenstance and coincidence, then they became institutionalized when people (companies and utilities) simultaneously realized that they were an economic and reliability improvement.

Think of the original road Ben Franklin laid out from Philadelphia to Boston - he needed a road to operate his new postal carriers from. So as he reached each town, he conencted to the existing "main street" because it was there, it was "straight" and most of the businesses were on that road. It also went past the courthouse or commons (because they "were" the middle of the town). At the other end of the town, he faced wilderness and farms again, so he started building a new road to the next town. To avoid farms, he went around the edges of each field.

So this road became institutionalized, and all sideways country roads started connecting to Ben Franklin's road "becasue it was there". Now, we have US 1, and it still goes right through "downtown" everything - the slowest possible way to go - becuase you can't move the road anymore. But everything else along the seaboard counties connects to it, becuase nothing else goes through.

Same thing happended to the grid - it started regionally as a backup, then went statewide )or larger) as companies realized that they could sell power through the same lines, then went nationally as power transmission became more efficient.

Now, they can't get away from it. But nobody "designed" it specifically - and that's part of th danger. The controls and communication is "embedded" inside in thousands of make-do and design to fix modes. The newer control rooms were not planned for year 2000 problems (nothing else is) so it's very difficult to separate the power plants and satellites and communications and telephones switchgear and power substations from the gird = they are using each other to keep each other running.

So whcih artery and which vein do you want your doctor to remove to test if you can still run a sprint? They are all critical to each other, and to the performance of the system (us). (And Canada too...8<))

So, the only satisfactory test is an "all up" test of the whole thing in real time with a simulated date applied to everything - and that will only tell us that the scenario being tested will either work or it won't.

How can you apply a test date to an embedded chip that you don't know is affected by the date switch? to an embedded chip you know "know" exists in a critcal system(s)?

-- Robert A. Cook, P.E. (cook.r@csaatl.com), October 19, 1998.

Richard:

I'm an engineer with experience in power distribution equipment. I designed transformers, HV and LV switchgear, LI switches, substations etc,. for the utility, industrial and commercial projects from the US to Saudia Arabia to Alaska to China...

I'll try to make this concise.

You have asked a question that cannot be answered.

Mr Cook P.E. (Professional Engineer - an exceedingly difficult qualification to obtain) gave you the best answer you will ever get. Engineers always design with the opinion that the product will work. We test to verify and learn new phenomenon to apply and consider in new designs.

1. It is impossible to create a simulation or model for large electrical grids. The variables of embedded systems are random, not discrete ocurrances at known times.

2. The total variables of the system cannot be calculated or even defined.

3. What does it matter if 36.7% of engineers are of the "opinion" that embedded systems represent a "significant" threat to a utility? What if I told you 83.44% considered it a "highly significant" threat? What are you going to do with that information? Do you have a weighted-average formula where one percentage point of "embedded system" threat equals 1.6 points of SCADA threat and 1.4 points of external control voltage threat to nuclear generating facilities? Does each point above 50 equal how many bottles of water you are going to buy?

4. Richard, the people working at utilities use the same products the utility generates and distributes. They do not get up every day and think to themselves: "I'm going to go to work today and make unreasonable decisions and do the worst job I can."

5. Buy some water, some canned food, a manual can opener, a kerosene heater, a lantern, fuel, some blankets, a few books and if your power goes out...just remember that 63.543% of qualified engineers are "reasonably confident" in their opinion that you will not spend the rest of your life in the dark.

I do agree that this thread is going nowhere.

Rega

-- PNG (png@gol.com), October 20, 1998.

Thank you Robert and Rega for your informed replies.

-- Richard Dale (rdale@figroup.co.uk), October 20, 1998.

Budd Y. asked, "What happened in the big blackouts...?

What I remember reading in the papers at the time is pretty much what CNN reported in its reprise 20 years later:

"For many New Yorkers, the blackout of 1977 is a dark memory. It started on July 13 about 9 p.m. when lightening knocked out electricity in much of New York City, plunging millions of residents into darkness. Unlike a similar blackout in 1965 that was characterized by calm, the 1977 blackout erupted in chaos -- and terror.

"Mobs set fires, smashed windows and hauled away food, clothing and appliances."

See After 20 years

-- Tom Carey (tomcarey@mindspring.com), October 22, 1998.