How is the electric utility industry completing remediation so quickly? This is an easy one, and the answer is quite simple.

The longest phases in y2k are typically the inventory and assessment phases. For large electric utilities, most finished around the end of 1998 (with some minor continuation to ensure completeness through the other phases).

After inventory and y2k assessments, you basically know what you have on your hands, what the y2k bugs are and where. It was at this point the industry could see that most embedded system y2k bugs were minor in nature and effect.

Then the remediation phase - the bulk of software upgrades, hardware upgrades to fix minor y2k bugs, were scheduled for the Jan99 through June or July time frame, simply because you need time to order, install, modify, etc. A few items are carried out to later in the year due to outages, software development, etc.

But here is the key to why so much progress was made in a short time frame - all of this remediation was going on in PARALLEL. So in March, April, May, etc, you start completing a whole lot of y2k upgrades during a short time period.

The proof in this will be in the remedation percentages completed by the end of June. For nuclear, probably >90% will be done. The only things NOT done will be the ones awaiting outages, a very few longer software development projects (typically not mission critical).

Regards, FactFinder

-- Anonymous, June 04, 1999

Answers

You forgot to begin this with "FOR IMMEDIATE RELEASE".

-- Anonymous, June 04, 1999

Please explain why your post is so much at odds with CA 2000 White Paper, from California's Department of Information Technology.

Thank you.

-- Anonymous, June 04, 1999

Actually the reason is that far fewer things were found that had to be updated or changed than first thought. An example is relaying. Almost all of the new relays can be set to give you the time, day, year. But this plays no part in the relays operation. Its just a logging function, not an operation function. Similar situations were found in DFRs and SERs. In an SER the date might be wrong but the time function (which is all we really care about) was still accurate.

When you start of thinking you have thousands of items to look at, and then it only winds up being less than 100 you can precede rather quickly.

In answer to your question Lane. Its because the people who wrote that know as much about the power system and how it operates as you do.

-- Anonymous, June 04, 1999

In answer to your question Lane. Its because the people who wrote that know as much about the power system and how it operates as you do.

What? No mainframes? No A/R, A/P, P/R, G/L?

Why don't you try again, with a little less condescension this time?

-- Anonymous, June 04, 1999

No 286's, 386's and 486's either?

-- Anonymous, June 04, 1999

As a vendor doing remediation work, as well as IV&V (Independent Verification and Validation), I can tell you that no, the assesment and inventory phases are usually only 20-40% of the entire process.

Yes, remediation can be done in parallel. However, many times when a large project is done in parallel, the end result is a mess rather than good code. Too many fingers in the pot so to speak.

-- Anonymous, June 04, 1999

Hugh:

You mention the prospect of "messing up good code." What aspect of remediation are you involved in? Mainframe? Embedded systems? I'm presuming not PCs.

What I'm trying to establish is whether there is a different timeline for mainframe systems, as discussed in the CA White Paper, and embedded systems, which I believe FactFinder is referring to.

I have heard that the metrics in fixing mainframes in considerably different than that of embedded systems. Some contend that embeddeds it's essentially inverted from the "20-40%" figure you mention. In other words, Inventory/Assessment - 60%, Remediaton/Testing -40%.

Can you, Factfinder, or anyone else offer commentary on this based on their personal experiences? Thank you.

-- Anonymous, June 04, 1999

You are right Lane, they don't. Walk into any substation. They don't run on mainframes, 286, 386,486,Pentiums, Apples, or even Linux. Some of the newer ones do use PC's as RTUs for control. And don't you think that would be one of the first things checked? But that doesn't mean that they are needed, or have anything to do with other stuff such relaying. As for SCADA. That runs on different systems depending on the vendor and vintage. Try PDP11's maybe.

As for the "condescension". Heck, you deserve it. You didn't have any problem writing "You forgot to begin this with "FOR IMMEDIATE RELEASE"." Or is it only funny when you do it?

The problem is while you may understand some coding you don't understand (and greatly overestimate the amount of computers needed) to run a power system in general and each substation in particular. There aren't a zillion things to check out in terms of Y2K. You are entitled you your opinions but when it's obvious you don't know what you are talking about (and you don't) you are not entitled to be treated as some grand po-bah who knows all. Especially when it's obvious you don't know much if anything about the way it all works and comes together. If you want to learn, ask questions. If you want to make smart remarks, expect them back.

I will give you some clues:

1) Dont confuse the business side of the house with the keeping the lights on side of the house. Different breed of people, different knowledge, different way of looking at things. 2) Dont confuse the equipment needed to make the corporation work and to do things like power flow studies with whats needed to keep the juice flowing to your computer. Different stuff all together. 3) Dont think because you understand how a computer network works you understand how the power grid works. A network is not a grid, so to speak. 4) Dont confuse using computers with needing them. I could walk into any substation in this country. Look at the prints, walk out into the yard and open or close any breaker with a piece of wire. Not brag. Just knowledge and experience. Might need two wires under special circumstances. Knowing computers doesnt give you any special insight into all things technical out of your field. In fact you are more likely to jump to erroneous conclusions and make a fool of yourself. Just because you know how A works doesnt mean you know a thing about B, much less X,Y, & Z.

-- Anonymous, June 04, 1999

FactFinder and Engineer,

You seem to have a case when you say that hardware problems are different than software. I can also attest to the fact that there seems to be little communication between the system and business sides of the utility industry (as little of it as I have seen). The thing that hasn't been well established to me is if one can affect the other.

Doesn't the business need the system, and the system need the business? The way you talk makes me think that the system doesn't really need the business. This does not seem likely to me. This is the thing I was talking to Bonnie about earlier. I tend to think that the inability to take money (billing) or the inability to deliver power (scheduling) or the inability to pay workers (payroll) may actually trouble the power system in a very serious way.

I think that we should talk about this.

Other questions I have are: Has your organization checked every chip that calculates with a date? Has your organization located every critical path? Has your organization located every chip in these critical paths that handles? Are you saying that you know for a fact that your organization's electrical system does not rely on any mainframes for the delivery of power? Is system testing at your organization completed?

If we could have the answer these questions then I think that risk could be managed more effectively.

-- Anonymous, June 04, 1999

Let's see Y2K comes and no one gets paid for a couple of weeks so in a snit everyone opens all of the brakers and goes home. To what? People who work for power companies tend to live in the same area. Wouldn't you call that cutting off your nose to spite your face? Especially in January! It won't happen. Several months ago about two weeks ofmy vacation time went poof. I came to work until it got un- poofed. I didn't go home and stew. Things happen all the time. Dealing with it is called acting like an adult.

Why would there be no billing? The billing results go to the business side of the house. The equipment is run the "lights" side of the house. All the meters I know of are, or have been made Y2K OK. And most had no problem. I think one had a leap year problem. But so what? If it wasn't fixed you get a day of free power. Meters don't control the power. The biggest problem was with the master and that was mostly because it was so old. They replaced it about a year or so ago. It works.

Scheduling? That's done on hourly schedules and changes both before and after the fact are done all the time. Scheduling is what you should be doing. What happens if a plane flies into a line? Wouldn't that throw the schedule off? Wouldn't it have to be corrected? Schedules aren't rigid dictum that have to be adhered to or the system falls apart. Think guide lines. You always have to be prepared for the unexpected. Transformer fire, generator failure, line down.

As the checking out everything. Are we back to that old saw again? Just because something has a date in it, doesn't mean it does anything except tell you what the date is. The idea that every thing has a chip in it (that has a date) is just not true. If you can give me some real examples of what you are writing about I'll listen but it's the same stuff that keeps going around that, really doesn't make any sense. The idea that everything has a chip in it isn't true.

There is some mainframe stuff (checked a long time ago) but it doesn't automaticlly control anything. It doesn't open or close breakers without a human doing the operation. There is some AGC (tested and checked) but again that can be used or not used. It gets glitches in it from time to time and they take it off line. I belive the present plan is to run some plants with it and to keep others off it, as a just in case. Computers are a help. They provide information as to what is going on and you use them to control some things remotely. But it's still someone telling the computer what to do. And as I said there is a lot less of this stuff than you think. So unless you can come up with specifics rather than what if's, I'm out of here.

-- Anonymous, June 04, 1999

) Dont confuse the business side of the house with the keeping the lights on side of the house. Different breed of people, different knowledge, different way of looking at things.

You Betcha Boy. I'm not confusing them, thank you very much. You seem to be asserting that the same rules that apply to all large companies don't apply to utilities because all that matters is the actual power production itself.

2) Dont confuse the equipment needed to make the corporation work and to do things like power flow studies with whats needed to keep the juice flowing to your computer. Different stuff all together.

See previous, thank you very much.

3) Dont think because you understand how a computer network works you understand how the power grid works. A network is not a grid, so to speak.

See previous, thank you very much.

4) Dont confuse using computers with needing them. I could walk into any substation in this country. Look at the prints, walk out into the yard and open or close any breaker with a piece of wire. Not brag. Just knowledge and experience. Might need two wires under special circumstances.

Walk on any lakes latey? And was that using one leg or two?

Knowing computers doesnt give you any special insight into all things technical out of your field. In fact you are more likely to jump to erroneous conclusions and make a fool of yourself. Just because you know how A works doesnt mean you know a thing about B, much less X,Y, & Z.

I see you're still having trouble with that little condescension thingy.

I felt a lot better about the utilities before I started to read you "professionals" on this forum. Thanks a lot....

-- Anonymous, June 07, 1999

Engineer, and Lane:

Please don't go...

Engineer:

The reason that I am asking questions is to find out if there is a problem. If there isn't one, then we will all know. I think that you seem to have some good arguments, but I want to understand. It seems to me you may have some bad ones (just like I believe I have had)

You asked about specifics. I have worked as a computer operator and watched as an entire business was reduced just about to a standstill while we worked staggered shifts trying to get a crashed computer to come up again. What happened was that there was a data problem on one of critical hard drives which corrupted the system so badly that we could not start it properly. We went to the back-ups and found that they had not run correctly for a while. (so much for contingency). We had to load from fragments of backups if I recall correctly. This happened more than once.

Going through experiences like this make me concerned for my own family and make me feel responsible to help in any way I can.

Unlike many hardware systems. Software systems are largely customized. Hardware has become very component oriented. An effort is being made by some organizations to become component minded in software, but the tools and technologies are still rather young. This lack of component oriented technology in software along with the non-physical nature and flexibility of software development (and probably quite a few other things) have helped us to create very unique, complex, and fragile systems that people (including the programmers themselves!) can barely understand. Some systems appear to be so complex that a 10 year project would not be long enough to reverse engineer the system enough to figure out how to fix it's critical Y2K problems.

I also want to mention that I realize that people may try to work without pay if they thought it would do them some kind of good in the long run. The problem is that employees have bills of their own that probably won't wait. When things get repossessed, or the threat of getting put out onto the street becomes real, the employee will be concerned for themselves and not for their employers who don't pay them. Wouldn't you agree?

-- Anonymous, June 07, 1999

As for the "condescension". Heck, you deserve it. You didn't have any problem writing "You forgot to begin this with "FOR IMMEDIATE RELEASE"." Or is it only funny when you do it?

Funny?

If the utility industry professionals writing here don't want to be taken for Public Relations personnel, they should quit writing the way PR personnel write.

-- Anonymous, June 08, 1999

I am sitting in the control room of a power station right at this moment. Yesterday we took our SCADA off line for a Y2K upgrade, and we will be operating the station manually for the next two weeks. All 4 generators are still running each day, and we are shutting down to 1 generator each night.

No additional staff are required, and as the generators are completely independant of the business network, there is absolutely no requirement for any computer equipment to run the station.

I can completely confirm that everything Engineer said about being to operate a substation just by looking at the single line diagrams is correct. In fact it is even easier to open most breakers than he has suggested.

So overall I belive that "YES" the lights will stay on. And you can take this from someone who has been in the Electricity generation and distribution business for over 20 years.

Malcolm

-- Anonymous, June 08, 1999

To Engineer and Lane Core Jr.

My husband and I read your comments to each other with interest. We aren't in a position to speak from first hand knowledge on the issues of computer programming or generation and distribution of electricity.

What we do know is that we are responsible for the welfare of a number of loved ones. Since there is no precedent for problems that may come with the year 2000, we feel the responsible thing to do is to prepare for the worst and hope for the best. We've only been doing this since January, but money that would normally go toward entertainment, cosmetic repairs to our home, and other non-essentials has been poured into canned food, bottled drinking water, medical supplies (including a medical field kit and books on how to use it), candles, matches, non-hybrid seeds and everything else we could think of for the comfort and care of our families, right down to the campfire popcorn popper. If Y2K doesn't change our current cushy way of life we'll have a lot of cool gifts to give for Christmas 2000.

However, I would like to add that one of our local news stations did what they called "Millenium Monday," and on each of their newspots throughout the day they talked with spokespeople in various fields to provide us with information on this very important issue. The results were as follows:

AEP (electric) and the water company: Prepare for at least three days to one week of outages. The newscast said representatives from the utilities will be in a bunker at the National Guard (explained below) in case of problems.

Banking: Huntington Bank in our town admitted to not being ready and said they will assist customers in any withdrawals they wish to make, but also advised that you hide it well if your do. They also advise keeping the last three banking statements and getting credit reports, etc. in case of individual glitches rather than an overall crash. I'm thinking of changing to their bank because of their honesty.

Prescriptions: Many doctors are advising their patients to get a three month supply of their medications. However, a spokesman from a pharmaceutical company said if we all do that, they won't be able to keep up with demand.

The Ohio National Guard has been preparing for four years, and in a newscast said they have a bunker twenty feet underground built for the occasion that is independantly powered. The camera went inside and showed stacks of boxes of supplies that were several feet high. The ONG spokesman said they believe their main functions will be medical and peacekeeping.

If all of the above feel it's wise to prepare, who are we to say don't worry about it?

Our battle cry has become "don't be scared, be prepared."

If we are prepared and nothing unusal happens, we've lost nothing. If we aren't prepared and the worst happens, or even moderate problems occur, shame on us for not protecting those we are responsible for.

-- Anonymous, June 12, 1999

Lane Core exchanged blows with the "Engineer" and after about 5 rounds, I thought Lane was the victim of a TKO. The engineer used good solid facts and not a lot of rhetoric. He was backed up by another engineer who also came across as credible. If this had been a TV debate, people would have turned off their sets saying there is no problem in the elctric industry.

I guess I must have missed the post that explained how the business side is either (a) unimportant or (b) already remediated.

Hey. I did not say, nor did I mean to imply, nor do I think that the business side is immediately critically necessary to the generation, transmission, or distribution of electricity. Okay? Can we say that's a GIVEN? How nice....

Now, is the business side important or not? If not, why not? Or is it already done?

In other words, remediating the production side is only part of the story -- PRESS RELEASES implying otherwise notwithstanding.

-- Anonymous, June 14, 1999

Sorry I got a bit snippy in that last post.

-- Anonymous, June 14, 1999

Now, is the business side important or not? If not, why not? Or is it already done?

the business side is *very* important, unless, of course, the vendors and staff do not require payment... than it is a moot point.

i do not know of many businesses that can go *one* week without the revenue necessary to keep the business going.

if there are no bills going out to the customer, or the bills are grossly incorrect, there will be *no* money coming in the door.

if there is no money coming in the door, how will the electric companies pay the costs of fuel, necessary to run the plants, pay the costs of trains, necessary to transport fuel to the plants, pay staff, necessary to man the plants?

if the electric companies cannot afford to buy fuel to run the plants how will they provide electricity?

and if they cannot pay the fuel distributor how will that distributor be able to afford more fuel and pay his staff and overhead?

and if the electric companies cannot pay the transportation costs how will the trains companies pay to buy more fuel to run their equipment, pay their overhead, or pay the staff to man the trains?

and if the electric companies cannot afford to pay their staff how will they man the plant?

and if the staff receives no 'paycheck' how will they pay their mortgage, provide fuel, food, electricity, etc., for themselves and their families?

the *heart* of any business is the accounts receivable... the a/r provides the data necessary to collect the revenues charged for goods provided or services rendered.

so yes, the business end of *any* commercial operation is *very* important.

-- Anonymous, June 14, 1999

To Those Trying To Decide On Preparedness:

I'll make these my last comments since they may not really fit this forum. If you're new to Y2K preparedness awareness and you're a little scared and very confused, please read my comments dated 6-12-99 and understand that, as I said, we're preparing for the worst and hoping for the best. To do so you must to listen to what you're being fed by your state or local utilities, banks, etc. and realize you're being given it in small bites so that you won't panic. No one really knows for sure what will happen.

Does the fact that my husband and I are stockpiling non-hybrid seeds sound like we're preparing for a week? If you're just starting you have a lot of catching up to do because if you get paid every two weeks and are living from paycheck to paycheck, you have about 12 paychecks left to draw on for supplies. If you have a nest egg small or large you may want to consider investing some into the future by becoming prepared now. And remember, the longer you wait, the more tuned-in to the situation others will be, and the prices of supplies will go up.

If you have limited funds, consider buying cases (12 each case) of canned goods. At Aldi's and some other discount food stores you can get a case for about $3.20. If you get potatoes, green beans and tomatoes and dump them into a pot with their own liquid and maybe add a few beef bouillion cubes you have a stew that can feed several people. Crackers are cheap at the above places. Add them to fill up stomachs and make the stew more palatable. Pastas and rice store well and don't cost much, nor does flour. Consider a woodburning stove with a flat top for heating and cooking-we bought a used one. If that's not feasible buy a simple grill and use wood or whatever you have for fuel to cook outdoors. If you feel you won't be able to afford or have room to set up a heat source, dress in layers and live in one room that you shut off with doors or with whatever you can nail or hang up to cover the doors to other rooms. You'd be surprised how much heat a few candles or oil lamps put out in an enclosed room; it's not like having a furnace but you can survive. Start collecting containers now to burn candles in so they won't be tipped over by children, and spend the eight or ten dollars needed for fire extinquishers. Give up the cigarette and pop or beer or Big Mac money if you have to. We've used flea markets to find lanterns and even enamel bedpans (we have four terminally ill loved ones, including a 19 year old with cystic fibrosis and multiple handicaps and a 70 year old mom with leukemia).

Anything can be washed and disinfected. We also found bikes and portable toilets. Bleach at the above stores is really cheap-get straight bleach with no fragrances and use 4 drops to a gallon of water in a clean container that you can seal; add to plain tap water and you have good drinking water for up to six months. Plain bleach is also a good disinfectant. Vinegar and baking soda have many uses, and now is a good time to learn about medicinal herbs and grow them and buy some standardized tablet forms at almost any grocery. Valerian Root is labled as a dietary aid, but is a great sedative. Talk to your doctor if you want to try it. For us it's proving to be safe and is an effective sleep aid.

We've tested various candle styles for longevity, we've tested recipes and herbs. This summer is a good time to learn to can food, even if you just start with a few jars of store bought tomatoes or peaches or green beans. We've logged onto countless websites for survival information, and have printed it off to avoid paying for books on canning and preserving food and many other issues. We're having fun with it, and if the actual Y2K comes in like a lamb instead of a lion, we'll have aquired a lot of new skills.

Still scared and confused? Well cut it out and think! What else will you need and where can you get it cheap or free? Go through your house and put every half empty bottle of shampoo, every saved little soap from Motel 8, every safety pin, every old sweater into your Y2K preparedness pile. Now is not the time to make a donation to Goodwill-if that sweater doesn't fit you it may fit someone else. Don't think fashion over warmth. And remember, there's strength in numbers so get family and friends involved. Have a New Years Party at the home where you plan to live without during the power outages whether it's for a day or a year and have the people who are going to live with you come to the party prepared to stay if necessary. Remember that those who aren't prepared may try to take what you have when they panic, so have a defense plan in mind.

Keep your newspapers and the junk mail that clutters your mailbox every day. It can become fuel to burn if you roll it tightly or toilet paper if needed. Save every plastic bag (don't use it for bathroom wastes, the stuff will brew, the bag will disintegrate and you'll have a mess. Buy a shovel.) If we end up with the lights on you can donate it all to a recyling center later.

These suggestions are just to show you that if you start now you can do it. Your family members and friends may laugh at you. Ours did. One of my sons is a police officer in a very violent area. In August his department is required to take Y2K training that will include learning how to shoot from moving vehicles. A National Guard friend has indicated that we may want to be in whatever county we plan to live the year 2000 in before December 31. Schedule vacation now if you're thinking of heading to someone's country or small town home, which isn't a bad idea.

Type "Y2K" or "Survival" into the search field and you'll find lots of detailed instructions on things you need to know. And don't forget that Christ won't be changed by power outages, and is the greatest source of power and strength available to us.

The odds are against my family because of our ill parents and nephew. But we're going to go into the year 2000 prepared and together and we'll handle whatever comes to us the same way. Others have done it under worse circumstances in the past; we can too. And if we can, you can.

Dee Kent

-- Anonymous, June 16, 1999