Last month (Kevin, maybe you can find the thread, I'm beat), Milne first, with me bringing up the rear, stalwartly defended the fact that the NERC figures entailed absorption of systems that didn't need remediation into the "completed" category, thus skewing the data and making it appear that the industry was further along.

As usual, we took a bit of a beating, even from Rick Cowles' superb board-mates ....

This just in from Rick's forum:

"Hello CPSR -

I have been on your distribution list for about 10 months and have very much appreciated the insights offered by your members. I am an independent IS Consultant with 20+ years in the trade, mostly in the management of large projects, having run a 300+ staff consulting group at one point in my career with contracts up to $25 million.

A recent assigment for a local hospital involved researching the "facts" about what has been coming out of NERC, and Washington in general regarding the electric industry. Of particular interest to this client is the likelhood of disruption to the Northeast power industry. After spending about a week crawling inside the NERC website, reprots and spreadsheets, I have come to some very disturbing conclusions. The "percent complete" methodology followed by NERC follows no standard project accounting method I've experienced. It is misleading to the point of absurdity.

To excerpt from my report for this hospital client:

Percent Complete vs. Percent Done

The NERC reports rely on an average of the percent complete reported by the individual participants and the average estimated completion date. As of November 30, 1998 these averages for the three major tasks NERC has chosen to report are:

Task Ave. Estimated Completion Date Ave. Percent Complete Inventory 8/25/98 96 Assessment 11/16/98 82 Remediation/Testing 6/6/99 44

The averages are calculated as the total percent complete or date divided by the number of respondents. This number also combines all components of the electric industry across all geographic regions such that a generating plant in Idaho that is ahead of schedule will offset a distribution company in Maine that is behind. As discussed earlier in this report, there are fewer alternatives in the transmission and distribution components and that progress in one component does not replace slippage in another.

The average dates reported are also misleading. If the goal is remediation of all mission critical systems by a certain date, then monitoring the average completion date is inconclusive in monitoring progress. Every day ahead of schedule on one component should not be used to offset slippage in another component.

The meaning of these averages is further confused by some specific instructions from NERC. The spreadsheet states:

% Complete - Report as amount of work completed in each phase divided by total amount of work to do in that phase.If no remediation and testing is required in an area that was inventoried and assessed, then show remediation and testing as 100% complete.

This instruction has the effect of overstating the percent complete of a participant in the remediation/testing task. Percent complete as used in the NERC reports is the percentage of systems that have been tested,not the percentage of the Y2K work that has been accomplished. There is a major difference between the two. For example, if a company has 20 systems and 10 of them did not require any remediation, they would report 50 percent complete. This is extremely misleading as it implies that the 50 percent remaining requires the same effort as the work completed.In actuality, no inference about the size, scope or schedule for the remaining work can be made. The NERC November report states that 44 percent (on average) of the systems have been tested, it is not necessarily true that the remaining 56 percent will require the same effort or can be completed at the same pace.

It is likely that the remaining work will require more effort and resources than what has been reported to date, as many of the completed systems required little or no remediation.

The NERC spreadsheets do not contain any indication of the number of systems that will require remediation, or the number of components that have been assessed, determined to be non-compliant and must be replaced.

There is no estimate of the scope of the work left to be done in any of the NERC reports. Without this information the likelihood of the completion dates being met cannot be directly determined or even estimated.

(End snip)

I would very much appreciate comments from others in CPSR. Has anyone else look at this??? It looks like political spin at its worst. Richardson, Sec of Energy was quoted in the NYTimes "We can be cautiously optimistic about the prospects for the industry meeting its Y2K challenge'' and that "tests and repairs are now more than half done".

More than 1/2 done??? The details in the NERC report simply do not support this.

Rich Hawkins Kingston Consulting, Inc. www.kingstonconsulting.com

from Rick Cowles (rcowles@waterw.com)

I'm not looking for a slap on the back, BTW. It was horrible news when Paul and I discussed it: it's still horrible news.

I see no reason not to assume it is typical of the way Y2K remediation is being estimated across other industries, though NERC has kindly allowed us the data by which to prove it in this case.

It is gross that the media and the government report this data but grosser still that the utility industry lets it be done. Of course, they know better.

As I always say, the compliance percentages are so meaningless, we could be ahead of the Y2K curve for all anyone knows. But don't bet your life on it. The name of this silly game is "percentage done."

I'll be waiting to see how the pollyannas on this NG try to spin this.

Hint: IT'S UNSPINNABLE.

(And thank you Rick and, most of all, Rich Hawkins, for confirming the facts).

-- BigDog (BigDog@duffer.com), February 07, 1999

Answers

I think this % has just kicked my 6.5 up the scale a little! I also would like to see more comments on this, Thanks Big... <):=

-- Sysman (y2kboard@yahoo.com), February 07, 1999.

Sic em Big Dog... good boy LOL Does ANYBODY know Don Merediths phone number?? Turn out the lights, the party's over, They say that all good times must end.........

-- Nikoli Krushev (doomsday@y2000.com), February 07, 1999.

This is the kind of thing that drives you bananas. Here's some of what has been reported so far:

1) NERC says 44% of remediation is completed. However, since it's possible that 44% of systems needed no work at all, this number may well mean that nobody has even started! You can't tell.

2) We read that TVA and some others are operating with clocks set ahead to 2000. Then we read that they aren't, really, after all. Then they are. What can this mean?

3) NERC reports that no known problems exist between the turbines and your wall plug. Then they say that they have found and 'addressed' these problems, whatever that means. There's no way to decipher from the NERC report a physical power generation problem from a billing problem.

4) This 44% 'average' number applies to nothing meaningful. It's like saying if you have one foot in the oven and one in the icebox, on the average you're comfortable. Meaningless composite numbers like this can only be considered a smokescreen.

So will we have power? I honestly believe NERC doesn't know. I wouldn't be surprised if the engineers at many plants didn't know either.

My only reasonable course of action is to assume I won't have any power and take action accordingly. How can anyone do otherwise, given the current information?

-- Flint (flintc@mindspring.com), February 07, 1999.

Good conclusion, Flint. Kinda outta character for you, ain't it?

-- Betta (betta@the.house), February 07, 1999.

Betta:

Bad news is bad news. Big Dog is right, you can't spin this. Why try? Only a fool would ignore it.

-- Flint (flintc@mindspring.com), February 07, 1999.

Garbage in, Garbage out. As a project manager, I have heard of this too, but not in the same industry (Power) - it's scary. I call it the "application shuffle" since the method is to change the applications classification subsequently reported to the mucky mucks to show 'progress' when in fact its just plain misclassified. This is another variation on what NERC is apparently doing. The key point is that by doing the application shuffle progress looks better on paper than it really is. Sad.

So to me, compliance reports can be a load of crap. The more pressure the mucky mucks are applying, the higher the chance for a shuffle. The only percentage that will end up counting is the percentage of lights that are still on in about 329 days from now.

-- Rob Michaels (sonofdust@net.com), February 07, 1999.

This is a good example of why this BB is so important. We get the facts backed up by facts. I want to thank everyone on this BB for their time, effort and contributions to keeping us informed. You may never know how many people you have helped to get prepared. Thank you again.

-- bardou (bardou@baloney.com), February 07, 1999.

I lurk around here from time to time, but y'all should learn to read, that is if you're not just trying to scam people.

The instructions read:

(% Complete - Report as amount of work completed in each phase divided by total amount of work to do in that phase. If no remediation and testing is required in an area that was inventoried and assessed, then show remediation and testing as 100% complete.)

This instruction applies when *no* work is required. In this case, they want to indicate 100%. What else would you put, 0%?

The example given of reporting 50% complete when there are 20 systems, 10 of which require no remediation, is completely bogus, and directly contradicts the instructions from NERC. From:

http://www.nerc.com/~y2k/assessment.html

Quote:

"A standard method for determining per cent complete follows. It is preferred that your responses be geared to % work done compared to total amount of work to be done. This would account for various activities having different amounts of effort. For example: if your inventory shows 100 devices with possible Y2K problems, and your assessment shows that only 2 have Y2K problems and one device has been replaced with a Y2K ready device and the other still needs remediation, the per cent complete to report would be Inventory 100%, Assessment 100%, Remediation and Testing 50%."

So in your example, they are instructed to use 0%, *not* 50%.

And you say someone *paid* for this report?

AG

-- AG (AG@BFI.com), February 07, 1999.

Actually, The nerc report comgines the remediation and testing. Therefore, a company cannot report on remediation unless testing is completed. Makes it hard to get an accurate state of affairs on actual progress of remediation.

FLAMES, FLAMES, I KNOW!!!!!

-- Moore Dinty moore (not@thistime.com), February 07, 1999.

COMBINES

-- Moore Dinty moore (not@thistime.com), February 07, 1999.

After the NERC report came out, i made lenfthy post to this forum explaining the faulty methodology and why the nuymbers were utterly unreliable becuase they contained systems that did not need remediation or little work at all. It badly skewed the exrapolation of how much work remained to 'BE 'done based on how much 'HAD' been done.

If the company said it had spent half its budget and had half its systems done. They 'could' be on track. but id half of what was counted as being done ALREADY was compliant, then they spent half their budget on only 25% of the total systems. meaning that they had TWICE as much actual work to go as they had alraedy actually done. it also means that their remaining budget had to be DOUBLED, minimally.

No matter how you slice it, they are NOT getting it done and not but a handful of utilities will actually be done on time.

The vast majority of electric utilities are not capable of defying 30 years of software engineering history and tens of thousands of hours of IT metrics. Not possible. Not in this universe or any other.

The lights are not going to stay on. In places yes, but overall, not a prayer.

When utilities start to fail the damage that is caused to the distribution and transmission systems will be extreme. It will NOT be something that they can put back together again as if it was a fallen branch. It will be grid wide.

How many time do you need to see the figures skewed by koskinen dropping mission critical systems or the NERC allowing 'remediated systems' that needed NO work to be included in work that they did, before you realize that you are being lied to.

Apparantly, even flint's comatose eyes flickered there for a second. Almost like his life support system stayed on.

So long suckers.

-- Paul Milne (fedinfo@halifax.com), February 07, 1999.

Gee, Paul, glad to see ya!

That the numbers may be badly skewed is probably true. But which way?

The original post claimed the numbers may be *over*-estimating the overall readiness of the electric companies.

But the example was false, and completely contradicted the explicit instructions of NERC.

In fact, the report may be *under*-estimating the overall readiness. As the example from NERC illsutrates, a company may have 100 systems, 98 of which are already compliant, with 1 remediated and tested and 1 not. They would report 50% on the spreadsheet, but would in fact be 99% complete.

Not suggesting the numbers are this skewed. But would tend to think they are *under*-estimating the overall readiness, based on the instructions.

Have a nice day, Paul.

AG

-- AG (AG@BFI.com), February 07, 1999.

Now, when are they going to do a "full up" integrated systems test of distribution system?

How many power plants are there Paul? (I remember you quoting 9000 before ? Seems like "30 per day" needed to report "remediation and testing complete" inorder to finish on time (based on or about a mid-January date)

How many "grid control stations" are there between all power companies? Where are they? How many are complete testing? Who is responsible for them Who is looking at their progress?

How much oil/nat gas is stored at each fossil palnt (Coal seems to average as much as three weeks, as little as 4-5 days.)

Last I heard, only two plants (the two in central Illinois ?) had actually run with "clocks ahead" - and that only after some pretty intensive repairs were made.

It appears at lest one distribution region in the northwest has cleared their central station to the pint where only substation problems may result. Good - but a couple of weeks ago, San Francisco penisula lost power when one substation failed.

Etc.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 07, 1999.

The San Francisco Substation failed due to human error, not computer error.

-- justthefacts (justhefacts.@jack.com), February 07, 1999.

Robert:

To answer one of your questions partially, the NERC reported:

"Of particular interest are the results of integrated tests involving the entire power station. More than 40 units at more than a dozen utilities have been tested while operating on-line and producing power. These tests consist of simultaneously moving as many systems and components as possible forward or backward to various critical dates. These tests require an extraordinary level of preparation and coordination to ensure the safety of all systems and that the impact to the electric system would be minimal should a unit trip during the test."

I only know of links on the net to 18 of them. 10 at Avista:

http://www.wwpco.com/y2k/y2k_qna.asp

And 8 at TVA:

http://www.tva.gov/y2k/progress.htm

AG

-- (AG@BFI.com), February 07, 1999.

There are arguments about that - that is, the fundemental reason for the substation being repaired (and thus subject to human error) was Y2K-related replacement of a part.

My point was that one substation failed - due to human error, as you pointed out - and this tripped off an entire city area (many municipalities).

One satellite failure knocked out 3/4 the pagers in the US. Its the system - I would prefer that it were held together by ticker toys and duct tape - at least then people would have half a chance of repairing it.

Now, what kind of reliability do you think we will get from an entire electric grid being run by radios and pickup trucks running from substation to substation masnned by overworked, dead-tired individuals who've been stressed by continual work for two days? Four days? Ten days? A month?

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 07, 1999.

Wasn't it only Friday that we heard the idiot telling us of 99.987% ?? I wonder what this will do to HIS math??

-- Sysman (y2kboard@yahoo.com), February 07, 1999.

This is so disheartening! This thread started on the right path with intelligent, rational and civilized analyses from BD's post, then Milne came along and changed the tone to put everyone on emotional defensive, which mucks-up everything.

Please people, be aware of this and simply ignore Milne's wild-eyed personality and his personal attacks, and continue to address the issue and maintain a rational, unemotional thread.

I can see for myself that the NERC report is misleading, but I don't have the technical expertise or the jargon to express what I see in it. It is frustrating, but I sense that it is even more frustrating for those trying to understand what it all means (newbies to Y2K, skeptic polyannas etc.) when emotions and flames muddy up everything.

-- Chris (catsy@pond.com), February 07, 1999.

Let's start a campaign to exterminate rats and squirrels. They cause more power outages and damage to wires than human error. Sorry Rocket J Squirrel, your now Brunswick Stew!

-- gourmetcheck (gourmetchef@pot.com), February 07, 1999.

I politely disagree Chris - paul responded "with vigor" but promptly and appropriately - and the replies to what I asked were exactly what I wanted - not what I wanted to hear, mind you, but with the specific numbers that I needed to know.

As of now, only forty power plants have been tested - partially tested, granted, but at least they were tested. These tests were "carefully" set up (which implies not complete - been there - done plant testing under wierd "preset" conditions - it ain't like operations after a scram or during a power plant recovery!), and only tested "several specific dates" (assume Jan 01, Feb 28, Feb 29, others ?), and were not complete runs in that everything was started/shut down/restarted on all crew schedules. At least though they were partially tested.

So when are the rest scheduled? Again - what about the distribution controls? Power to the "fenceline" is the easiest part of the problem, particularly if only forty have been checked so far.

I've built power plants, tested power plants, and re-tested power plants after rebuilding them - and they ain't done yet if only forty (of 9000 ?) are tested.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 07, 1999.

Big Dog, Now that's good hunting!!!!

-- Watchful (seethesea@msn.com), February 08, 1999.

Chris, why does Paul Milne make your underwear roll up into knots? Is it because you are squirming from fear and the truth that you don't want to face the facts? The truth does hurt, truth is truth and it only hurts for a little while. I really think that if it were served to you on a silver platter with raspberries and whip cream, you would still poo-poo it.

-- ~~ (~~@~~.com), February 08, 1999.

Plus to add to your analysis Robert, there is still "burned in" code on millions of chips that do not allow you to change dates internally. It's one thing to roll a date forward on a DCS system, but quite another to get inside the embedded coding on a chip that was added in the manufacturing process. I never hear this discussed. All I hear is "we've been running this plant off-line with the date rolled forward and everything's fine". You can roll dates forward all you want in PLC's, DCS's and other control systems, but you're still not rolling the date forward in each individual embedded hardwired chip in the system.

-- James Chancellor, P.E. (publicworks1@bluebonnet.net), February 08, 1999.

And that Mr. Chancellor, is the big if that we will find the answer to when rollovers happen throughout an unkown period starting at an unknown time.

The concept that we would test them all beforehand gave way to testing only the mission-critical systems. That gave way to the 'plan' that we will just have to test what we think we need and "wing" the rest... hoping for the best and fix-on-failure as fast, as we can, anything that stops working.

This is the plan for some of the biggest companies and infrastructure services in the world. A wing and a prayer.

Fix-on-failure is considered to be at least 100 times more cost efficient than testing.

-- PNG (png@gol.com), February 08, 1999.

IF you can fix-on-failure.

Granted, fix-on-failure will work - IF you can diagnose what's wrong. The result of the failures at the top level of the process is not the symptom that will tekll you what failed that CAUSED the outage or process failure. (All you know is: the process stopped, we had to abort the run, we can't startup, we can't align the instruments, we can debug the central control unit.)

The immediately of fix-on-fialure result is most likely not a conclusive nor immediately obvious result that you can then say: go replace "this thing" - assuming you can get hold of "this thing" in the first place.

Rather, I expect - again, based on first hand testing results under much more controlled conditions than a "panicked plant manager and CEO" will be seemlingly endless repeats of "Pull this one, and reinsert the other one. It might be the one. Shutdown again, turn the power off. Then we'll try it again. ......"

And multiple failures - each masking the first, second, or third layers of the original problems are even more tricky to figure out When you are not tired, discouraged, and running out of spare parts.

Because each failure might destroy good parts. Remember the steel mill - Australia I believe - that failed in early 1998: aftert hey thought they had repaired everything "....each system, one after the other, stopped.......until the whole plant was quiet." Molten steel does not "recover" gracefully" once allowed to cool inside complex machinery.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 08, 1999.

Sorry about the typo's there guys - 'tis late and there is much still to go tonight/this morning to get the boss off on the plane tommorrow.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 08, 1999.

There has been a lot of talk about "fix on failure". Who will be around to do all this fixing? By the time it gets to "fix on failure" the runs on banks, the panic for food, the realization this will be very bad will have sunk in. Are these "fixers" going work for nothing? I've always felt that these "fixers" were only interested in the almighty greenback. I think they'll be scrambling around with the other DGI's trying to get some prep in. I can't see them hanging out at the "Company" trying to "fix" the mess, especially if they realize they're not going to be paided. Just how many "fixers" like that do you think we have in America?

-- thinkIcan (thinkican@make.it), February 08, 1999.

Hopefully, all the 'fixers-on-failure' will also have been 'stockpilers' to some extent. That way, they'll have a few less things to worry about while doing their fix-it thing.

Oops, politically incorrect me, we've just been told it's 'needless and frivolous' to set aside any necessities. Never mind...

-- Nathan (nospam@all.com), February 08, 1999.

Here's what bothers me the most about the nuclear power plant issue...

This is a quote from a January 12, 1999 article from Newsbytes:

http://www.newsbytes.com/pubNews/124358.html

[begin snip]

However, an NEI statement said that the "majority of America's 103 nuclear power plants" still have not finished the assessment stage to determine which of their computer systems will be affected by the Year 2000 problem.

Specifically, NEI said most of the plants "have nearly completed the detailed assessments needed to pinpoint computer systems that might be affected by Y2K issues."

"Plants that have not completed this work have been asked to increase their efforts so that the industry can report on the Y2K status of all plants to the Nuclear Regulatory Commission by the agency's July 1, 1999, deadline," the NEI statement said.

[end snip]

My comment is that the reason for the July 1, 1999 deadline is to allow time to cool down any plants that aren't compliant and need to be shut down. The NEI is apparently hoping that mere assessment of nuke plants will be sufficient to convince the Nuclear Regulatory Commission to allow them to continue operating after July 1, 1999.

Since it does take months to cool and safely shut down a nuclear power plant, it's obvious that assessment, remediation AND testing need to be completed by July 1, 1999. This is an issue that will significantly increase Y2K awareness with the general public in the summer of 1999.

I started a thread about this last month at:

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000Npi

-- Kevin (mixesmusic@worldnet.att.net), February 08, 1999.

Please Kevin - don't get misled about the supposed "months" to cooldown power plants. Tain't so. they can be cooled down to room temperature in two-three days, and will remain that way - external power or no. We've covered this mathematically, operationally, and via nuclear physics analysis several times.

The margin is typicalof how the NRC and the industry approached things - set a deadline early enough to be certain it's right. To give every compliant plant time to continue running before the problems hit. Time to re-write procedures, check things, maintain the plant through regular cycles. Etc. Get parts. Plan things. Test things. Drill crew members. To give a margin if needed to certain plants that have shown they may need a waiver. Etc. To identify outside emergency procedures that may need updating.

All the stuff that conventional plants aren't doing that will tend to cause severe problems when they try to recover under emergency conditions from uncertain troubles and shutdowns.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), February 08, 1999.

Robert,

I knew it didn't take a whole five months to shut a nuclear plant down safely, but I didn't realize it could be done in days.

Point well taken. Thanks!

-- Kevin (mixesmusic@worldnet.att.net), February 08, 1999.

As I recall, 75 power plants generated about 50% of US power, and the total number of power generation facilities was on the order of 200. The remainder of the 7000+ are non-generating distributors. Which must also be compliant, of course.

-- Flint (flintc@mindspring.com), February 08, 1999.

Just an update. Found information on 12 other units that have been tested by Southern Co.:

ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/southern-co-presentation. ppt

It is a PowerPoint presentation. They plan on rolling over 8 more by May 1st.

AG

-- (AG@BFI.com), February 08, 1999.

I didn't know that AG's company reported to NERC??

BFI's website; AG's home

Ratchet me up another notch.

-- Jelly Bean (jelly@belly.com), February 08, 1999.

Does anyone in this forum know anything about how the state of Maine power companies are doing?I would realy like to know.I'm a GI preparing for a 9 with the help of my family.Ijust can not make heads or tails out of the date and % crap they put out.

-- Darlene (boomer@tdstelme.net), February 12, 1999.