and quite nicely, i might add.

this is the crux of the whole issue. reverting to manual and the risks attendant with same.

engineers may be technologically astute but they are not known for being particularly savy as students of human nature.

i have had people working for me most of my adult life and i have set up systems for approximately the same length of time... this is our achilles heel--reverting to manual.

this is what i especially fear in regards to the nukes.

the problems at tmi should have been easily handled, the fail-safes and the controls were all in place... operators made some real 'bad' judgment calls... and the rest is history.

remember, many things work out real well... in theory.

http://www.prorege.com/north/4997.html

-- Anonymous, June 14, 1999

Answers

No, Gary doesnt sum it up. Gary is just really really clueless about how it works.

1) SCADA doesnt control everything. In my company, for example, about 2/3 of the substations are on SCADA. The other 1/3 are not. So if you need SCADA to control everything how do those 1/3 work?

2) The statement that if you loose SCADA the transformers will fry is simply wrong. Transformers, Lines, Breakers, Bus, etc, and et.al. are all protected by relays. The relays are not dependent on SCADA to operate.

3) While Gary has this mantra about having to give operators (special) training to operate the system manually he is way off base. Operators have to operate the system manually all the time. When a crew goes to a substation to work on a piece of equipment it has to be taken out of service by an operator. As an aside, some companies have their crews also do the switching. If the breaker can be operated by SCADA it may be opened by the associated disconnects still have to be unlocked, opened, and relocked, by hand. In addition, if relays and Transfer Trips have to be switched in or out of service or bypassed, this is done by hand. In all of our stations there are written instructions on how to operate the station.

Every station that Ive ever been in, both in my companies and in other utilities, has a way to manually operate every piece of equipment in that station. If anything our operators have to receive special training to become SCADA operators. Not the other way around.

The only thing that Garys statements show is how woefully ignorant he is of how a utility operates.

-- Anonymous, June 14, 1999

Engineer ... You may be right about all that "manual dexterity" , BUT, I am reminded of the many westerns where the young, would-be gunfighter is taking lessons from the old pro on shooting bottles off the fence posts. Finally, after much practice , the young cowboy shoots them all off, NO misses, AND his draw is so fast he says to the old gunfighter ; " I'm ready now, maybe even for you ?" To which the old gunfighter replies , " Yes, your draw is fast; your aim is good ; BUT, you'll notice , the bottles AREN'T SHOOTING BACK !!! " My point ? Just like Three Mile Island, driving a car where you have read the manual, and going on stage to sing the song you practiced a hundred times , CAN YOU STAND THE PRESSURE WHEN THE CHIPS ARE DOWN ??? This WILL make every situation where lives are at stake a whole new ball game. Had it happen to me in the navy. I'm in CIC ( Combat Information Center ) , just out of radar school six weeks and we are called on by the skipper to give a course and speed to come to 500 yards astern of a carrier , which will turn into the wind in five minutes to launch planes . It's pitch dark outside ( we don't even have a porthole ) , the OTHER destroyer on the starbord side of the carrier will criss-cross with us to take position on the port side (astern) of the carrier; we must be on the starboard side , to pick any pilots that may "ditch" due to engine problems. The exectuive officer joins me ; I'm relived to see him ! The skipper calls down three times for the course and speed . My mouth is dry and it would take a crowbar to open it, I'm so scared, as I realize hundreds of mens lives are at stake, including my OWN ! The exec looks at me , and says in a wisper " What did YOU GET ?" I gave him my answers, and he says "That's what I got too ." He calls it up, we make the turns, and later I hear from one of the deck crew that a man on the bow was almost able to shake hands with a man on the stern of the carrier ! ALL the problems I solved in school AND aboard ship before that ment NOTHING at THAT MONENT IN TIME. If you haven't lived it , you won't understand it ! Eagle

-- Anonymous, June 14, 1999

hal,

that is precisely what i mean. when the heat is on it's a whole new ballgame.

you are the last man up... the bases are loaded, it's the bottom of the ninth, your team is 2 runs behind and the two guys before you just struck out.

so do you handle the pressure... or do you choke?

well, engineer. this here ain't no ball game... we are talking about people and their lives. senior citizens on life support, kids on dialysis and life support systems. people that live in verrry cold climates with fragile health. this is serious business.

... and yes, you could work for the best company on the planet... but guess what? there are thousands of electrical plants out there and 103 of them are nukes.

i have absolutely no doubt that you mean and believe everything that you are saying but i am telling you that there is a much bigger problem out there and it transcends your particular plant.

let me give you another analogy. when i was a kid i raced in several hill climbs. now i had a triumph spitfire, some of the guys had porsches, healeys, etc.

now from a performance point of view those porsches and healeys should have won hands down... but guess what? they didn't have what it takes going into those curves.

technologically those cars were far superior to mine... it just isn't all about technology.

-- Anonymous, June 14, 1999

We Can Run It All on Manual, No Problem, No Training, Just Like Always

-- Anonymous, June 14, 1999

Thanks marianne ! Guess what ? Never raced it , but I have a 70 MGB GT and it IS great on the curves ! Hal

-- Anonymous, June 14, 1999

ComEd, my electric provider, uses SCADA up to the substations.

After that, they do it manually.....sort of.

If any feeder line goes down, shorts out, whatever, AFTER a substation... they have NO IDEA there is anything wrong until a customer calls their "800" number to report a power outage.

Not only that, there is a threshold of calls that must be met before a repair order is issued!

So if I'm the only one who reports an outage for our whole area, nothing happens!

It used to be that outages caused by wind or rain or fallen tree limbs would last 45 minutes at the most.

Now the average is about 2 hours and counting.

Why?

When the power goes out, noone calls. they assume "someone else" knows about it and is dealing with it, during the last power outage, I polled some of my neighbors and none of them had called ComEd.

Partly because they thought it was someone else's job, partly because they all had cordless phones THAT DON'T WORK WHEN THE POWER IS OUT and they had no "normal" cheapo land-line phone. Some of them were just wandering around outside "waiting" for the power to come back on!

It looks like ComEd's "manual backup" on this end is failing.

they forgot to give out the instruction manual.

-- Anonymous, June 14, 1999

gosh lane,

gary must read rick's forum. he apparently doesn't agree with the engineer. i think the engineer just got hit by a train... the north special coming daily out of arkansas.

gary certainly can analyze and synthesize data... i guess that is what they teach you when you get a phd.

text from gary north's assessment:

My comment: I'd like to know what company employs this man, and I'd like to see its y2k compliance statement. For all I know, he is an engineer for the Hog Jowls, Alabama, Power and Bait Shop Company, Inc. What I'm wondering is if the power companies for cities of a million people and more can run their systems without computerized SCADA systems. For how long? At what price? With what reduction in output? He says that I don't understand his industry. Actually, I understand it quite well. It is not compliant. All the PR flak, all of the excuses, all of the "we know what we're doing" chest beating cannot overcome this simple fact. The industry is not compliant, and there is no way at this late date that it will reach compliance in 1999. That's the big picture.

Engineers don't like big pictures. They are paid to understand little pictures. The industry buys computers to deal with the big pictures. I have a view of engineers' thought processes. It goes like this: "This is a bolt. This is a nut." Excellent, but how can this company be run without compliant computers? "This is a piece of wire. It is top-quality wire." It looks just right to me. But, about your computers. "This is a switch. It goes on the wall over here. . . ." But, about the computers. "There is a formula. . . ." And so forth, until the day the lights go out. No SCADA? No problem. You read it here first. This is from Rick Cowles' site. end of text

-- Anonymous, June 14, 1999

Interesting. You guys seem to value the opinion of someone who is outside the industry more than inside because his take falls in line with your sentiment. But relying on North's assessment makes about as much sense as asking a butcher to do brain surgery because he's handy with a blade.

-- Anonymous, June 14, 1999

no, that is not true... what we are searching for is the truth. not spin, not everything is fine, not we didn't need those computers anyway, not we found nothing that will prevent the generating of electricity... the truth. some good news and some not so good but not this assinine babbling that we should take our non-industry aware selves home and sleep tight.

check out this thread before you come to any conclusions about why we may perceive those outside the industry a tad more than those in the industry. scroll down to engineer's post and then read bonnie's reply.

http://greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000wng

then ask yourself... what would you believe if you were in our shoes?

we are the public for whom the electrical industry provides all that wonderful electricity. we are the ones that pay the bills. we are called customers.

-- Anonymous, June 14, 1999

Yep,

I'm going to second Marianne's statement and then add this. We know who Gary North is. In fact we know a lot about him. His life is almost an open book. Personally, I don't know diddly about Engineer, beyond the "trust me" position. He doesn't identify his company, or publicly provide the bonafides. So he talks the talk. So does Koskinen. Engineer doesn't get my vote unless he talks real deep intelligent sense, and not make sweeping statements about *his* entire industry. Look at the post on CT utilities. They aren't sharing information. But Engineer wants us to accept that he has the pulse of the entire industry. Please. Give me a break. I was born at night, but not last night, as the saying goes.

-- Anonymous, June 15, 1999

Marianne, I don't believe that you need to worry about operators not being able to perform should the worse happen. It has been my experience that most operating errors occur more during routine functions than during urgent or emergency conditions.

Here in New Zealand we had a major cascade event in 1986 when I was newly qualified as a system controller. We lost supply to over half of the North Island due to a major fault in a substation, and every power station north of the center of the Island went off line. Only a single hydro station and our very old geothermal station in the south were able to keep some generators spinning. Among the generation stations lost was our gas fired Huntly station which is the largest in the country, and supplied almost a quarter of the countries needs. I'm mentioning this particular event because it is the closest that I can imagine to a major Y2k collapse. The reconnection of the entire grid took almost 4 hours, and it was all done manually. Two years later a similar incident occurred, affecting a much smaller area, but still taking out the Huntly station. By then we had SCADA, and the reconnection, this time using SCADA instead of manual operating, took just over five hours.

I am now a Production Controller for the second largest generation company in New Zealand, and a week ago we took the SCADA system for our largest hydro station off line during its Y2K upgrade. Since then the station has been operated completely manually, and there have been no problems reported. This is our newest station, and the last one to be upgraded. All the thermal stations were modified first, and have passed their tests OK.

Right through the remediation process, all of our operting staff have really enjoyed going back to manual control, "Hands on" as they call it, and its amazing to see how even a couple of weeks of "real operating" helps to raise their level of performace. Its so good that even after all of the Y2K issue is over we are considering running each station on manual for a week or two at a time just to keep everyone's proficiency up.

I'm not sure just what the situation is in USA with your nuke plants (we are nuke free over here), but we are preparing for possible disruption due to such things as communication failures. We are installing alternative communication systems that do not rely on any of the traditional networks, just in case.... :-)

Malcolm

-- Anonymous, June 15, 1999

I personally don't place much stock in old Gary. I believe that if we all woke up on 00/1/1 and nothing at all happened, his site would have a huge posting saying, "The World DID end, but the government is just lying to you so you don't realize it." What else should we expect from a man who sells gold? I'm not saying that any industry is compliant, but would Gary even Acknowledge the day that any one industry does become compliant?

-- Anonymous, June 15, 1999

Frank, Gary doesn't need me to defend him as the post above shows. But he doesn't sell gold. And secondly, Gary's whole point is that there is no way an industry can be compliant before the rollover. To do so would mean all of an industry's suppliers and vendors are also compliant. Get it?

-- Anonymous, June 15, 1999

marianne and engineer,

Gary North and his observations about the power industry are of great value precisely because he is not in the "industry". All of us, when we are exposed to something continually, on a regular basis become enured to it. Even the best food becomes cloying when eaten too often. An astutute outsider often notices things which insiders no longer notice. It is human nature. We, humans, would go crazy if we allowed everything to bother us all the time.

I have often worked on temporary contract jobs in various countries. Each time, I go in with fresh eyes and insight into problems and set about fixing them. The locals I work with both want and are exasperated by my "American" eyes. They are at the same time an asset and a pain in the asset, however, it stirs them out of their "comfort zone" and things end up working better. I find this pattern repeated on every job. In addition, when I come home to the US I look at it through "foreign" eyes and appreciate its assets and liabilities more.

So take Gary North's "outsider" views more to heart. He sees things that you can no longer see as an "insider", whether he knows how to fix them or not. His observations should be valued, appreciated and used to make you less comfortable with the status quo, move you out of your comfort zone, and make you not so complacent, to the advantage of all of us. We are depending on you "insiders" to make sure the problem doesn't happen.

-- Anonymous, June 15, 1999

Frank-

Where does Gary North sell gold?

also- maybe YOU could post the news when ONE INDUSTRY becomes Y2K compliant? that would be helpful, don't you think?

I look forward to your response.

-- Anonymous, June 15, 1999

I am now a Production Controller for the second largest generation company in New Zealand, and a week ago we took the SCADA system for our largest hydro station off line during its Y2K upgrade.

Malcolm, why was a "Y2K upgrade" necessary? Or wasn't it? If it wasn't necessary, why was it done?

-- Anonymous, June 15, 1999

Theres a lot to reply to here so Ill try and take it one at a time.

Actually Hal your analogy isnt very good. The western scene you are talking about is myth. The closest that comes to it is Bill Hickcocks shoot out with James Stillwell (sp?). Most of the time gunfights were ambushes and each side used as much cover as possible.

As for dealing with pressure thats what operators and dispatchers are trained to do. The analogy would be to a fireman. They arent out fighting fires all the time. A lot of time is spent in the station reading, working on equipment, training, even relaxing. Whats important is that they are well trained and ready when the bell sounds and they have to hop on the truck. Same with operators and dispatchers. They have retraining and review periodically. My point with Gary was he thinks that some kind of special training is needed come Y2K because no one knows how to do everything manually anymore. Nothing could be further from the truth. The special training is to operate the SCADA not the other way around. And you have to be a good operator (and bid for the position) before you become a SCADA operator.

If youll permit me a story: When I was a somewhat younger engineer I was putting in SCADA at a fairly important substation along with an older, more experienced lead engineer. The Operator on duty was just out of the apprentice program. Something happen when we were testing and another breaker (other than just the one we were testing) opened and we dropped a line.

This didnt put anyone in danger its just a blot on your copybook. You have to write a letter about your screw up and get grilled about it at an open meeting before a group of your peers, etc. So you try to avoid making stupid mistakes and making a fool of yourself. The two of us ran into the control room and started to look at the annunciator panel and wonder what was going on. We were having this debate and walking around arguing with each other. Finally the operator (who was junior to both of us in years and service) told us to get out of the control room. Told us to leave NOW. Guess what? We did. It was his bailiwick and he was the man in charge. After he got the station back together he let us back. We figured out that it hadnt been anything we did. Just by chance lightning had hit the line tripping the breaker at the same time we were testing SCADA on the first breaker. We had relay targets and an oscillographic record and when we tested the breaker again (without changing anything) it was OK.

So no, Im not worried Hal. He did what he was trained to do. Just like you did.

marianne: See Ricks comments in another thread about the Nukes. And as for you train comment: Yeah, I got hit by a train all right, HO gage.

Plonk: Not to sure if you mean Commonwealth Edison but anyway: The reason they need a number of calls it because if one person calls in they dont know if its just your house or the neighborhood. If you are working in your house and you trip the main circuit breaker (or blow fuses, if you still have them) how are they supposed to know its them or you? There are devices that will plug into your wall and make a telephone call if the power goes out for a certain amount of time. But even then unless they get X number of calls from the same neighborhood they wont do anything. They dont want to send out a crew (especially on overtime, on the weekend, at night) to find out you screwed up plugging in the Christmas lights. Guess what. The utility that services my house works the same way.

However Im sure that if you tell them that the lights are out over the entire neighborhood theyd send some one. If no one calls than call them (the company)and see if you can do work with them. No one here can solve that problem. If it gets really bad call a reporter at the local paper. As for the cordless phone, yup, you are right about that. Of course I have a couple of old rotary phones in my house (but than Im an engineer). You can still find them, as well as the push button tone ones. And used they are pretty cheap. Of course if you have a cell phone that should work in most of the cases.

As for who North is: His Ph.D. is in history. His history of predictions isnt very good. The advice he gave was the same hes giving now. It seems to me that if someone cried wolf youd start to wonder when the wolf didnt show up after a while. Dr. North is entitled to his opinion but hes not entitled to a free pass. Especially when it is clear that technically he doesnt know what hes talking about. Its one thing to make an incorrect assumption and have questions. Its another thing to cling to your beliefs about how something operates when its patently false. If someone believes that substations operate by witchcraft and that come Y2K all the spells will fail you really cant have a rational argument or debate on the subject. While Gary isnt quite that bad, its close. Words like SCADA and embedded chips , secondary clocks , etc. are being used as magic words. Invoked to summon up pictures of looming stygian darkness.

Everyone is entitled to their opinion but all opinions arent equal. If this were a medical forum and I was looking for a cure to something I would read the posts but the Dr. I would go to would be an MD. Hopefully some one with at least a 2.0 GPA and had been in the field for a while. Not a Ph.D. in history. No matter how insightful his comments are.

As for not posting under my real name. I thought about it. The reason Im not is that when I did occasionally send people in other lists posts under my real name I got a lot of junk back. I dont mean flames. I like to think I have a good sense of humor about that. But stuff telling me that Unity Point source machines (perpetual motion) were the way to go. Some guy who claimed to be a telecom engineer sent me a post about all of the embedded chips that were in PCBs and made sure all the poles opened at zero voltage. I politely told him that wasnt the way it works and I got a post back telling me that he knew this was the way it really worked and I should In Christian Brotherhood admit it and warn others. So no thanks. Id like to keep my nom-de-plume if you dont mind.

And I do have a real job. Writing all of this takes time. And Id like to get back to it. Im having problems with 9 across.

-- Anonymous, June 15, 1999

Lane, a Y2K upgrade was needed because the previous system used an outdated version of UNIX that the manufacturer was no longer able to support. We were unsure just what would happen on the rollover, but one function that would have caused problems was the date/time tagging of events. The latest 4800 events are available on screen, but anything older than that is automatically archived. With only a two digit year, any events occurring after 1/1/00 would appear to be older than any events in 1999, and would be archived before they were displayed.

So, an upgrade was required; The manufacturers of the system were not able to assure us that the operating system would work correctly; and some date/time functions would not work correctly.

We have now been 9 days without SCADA, and all is well. Our new SCADA system will be up and running within a week.

Malcolm.

-- Anonymous, June 15, 1999

engineer,

just wondering what your take is on the folloiwng:

1. on average, corporations in the usa have spent about 50% of their y2k budget to date. that's working on the problem for 3 - 5 years. our local power company, nimo, i believe has spent far less than that. and with 95% of companies STILL coming in with revised/increased y2k budgets, its not like they've overestimated their original ones. i dont get a very warm feeling from this. do you???

2. the one thing that REALLY bothers me though, is how pathetically little information is being released by power companies. have you read any of their y2k statements? their statements are all identical. they're saying practically nothing! (i know i know .. their lawyers dont want them to say much). my opinion is that - for the most part - the entire country is covering this thing up. its so obvious.

lou navarro not afraid to give him name

-- Anonymous, June 16, 1999

Engineer:"Plonk: Not to sure if you mean Commonwealth Edison but anyway:.."

No, I meant ComEd...they are no longer known as "Commonwealth Edison", they officially change their name to "ComEd".

Engineer: "However Im sure that if you tell them that the lights are out over the entire neighborhood theyd send some one."

I'll try that next time, provided I can get through to a Human.

I would rather that ComEd somehow informed all its customers that they were part of the FeedBack Loop.

Absolutely no one seems aware of this. (except the people reading this forum)

-- Anonymous, June 16, 1999

Thanks, Malcolm.

-- Anonymous, June 16, 1999