Again on Rick Cowles open forum in response to my "will anyone speak off the record".

Here's some off the record for you:

..These paragraphs are from a utility insider who attended a state- wide session on Y2K and embedded systems in electric utilities in a Mid-West state in 1997. I have confirmed most of what this individual relayed to me via email correspondence with others who also attended that session. He reported that a leading electric utility engineering firm official and a leading Y2K expert with a leading electric utility industry group told the utilities the following:

In the testing of two coal-fire power plants (which were currently offline and being used as "hot spares") for year 2000 compliance, the clocks were simultaneously rolled over to the year 2000, causing immediate plant failure. In an attempt to better understand the failure, the roll over test was repeated. In the second test, the plants again failed, but a different embedded controller was determined to be at fault. The roll over test was repeated a third time in hopes of replicating one of the previous failures. In this test, the plants failed from yet a different embedded controller. It was determined that this last failure would have caused a grid-wide failure had the plants been online. It took 13 days in order to restore the plants to working condition from the last failure.

OFF THE RECORD:

The discussions that took place in the meetings really scared the hell out of me. It seems that the power utilities that have not yet recognized the embedded systems problems are already too late. It was said to take about 21 months and $30-40 mil to make one generation plant compliant. The utility companies at this meeting expressed the opinion that compliance would not be possible due to budgetary and time constraints. "The only thing we can be certain about the year 2000 is that we won't be able to fix everything." The opinion was expressed that complete Y2K remediation is an insurmountable task, therefore utilities should just attempt to make the steps necessary to prove due diligence in the court of law.

According to this official, the money that all utilities report they are spending for Y2K is outrageously low. The problem is that the current tax accounting laws encourage the utilities to only publicize their software Y2K budget --because those costs have to be expensed out in that year, not depreciated over many years. Thus many utilities who are trying to be diligent are reportedly lumping the bulk of their Y2K equipment replacement and/or upgrade expenditures into normal equipment upgrade costs. Consequently, this data is hidden amongst non-Y2K costs and neither the public nor the regulators can distinguish if they are doing a first class job or a really cheapskate, risky one. I've talked with both Y2K tax attorneys and managers at utility companies and I know this is happening, to my great frustration.

I have also been told by managers working inside these utilities that they do not want their shareholders to panic over the full amount of money they are spending on Y2K upgrades. Personally, I would think educated shareholders would be more willing to hold onto their utility stock during the Y2K crisis if they knew the utilities were spending more on the problem, but apparently the utilities feel otherwise.

Consequently, it is impossible for an outside Y2K expert to grade these utilities on the quality of their Y2K work simply by reading their SEC reports and by hearing their Y2K presentations. Most of us studying the Y2K utility problem are totally in the dark and that includes the regulators. How frustrating! I do know that some utilities are reportedly taking a FOF approach (Fix On Failure) and that really concerns me due to the 13 day downtime incident illustrated above. (Yes, I've had a few Y2K utility project insiders write me and tell me the management is engaged in type testing or plan to include use of the FOF approach. This might actually be acceptable for non-mission critical equipment, but one Y2K project manager felt too much was being excluded from testing at his company.)...

by Roleigh Martin: ----------------- Note: this message was sent by a robot.

(I like that bit)

-- Richard Dale (rdale@figroup.co.uk), November 03, 1998

Answers

<>

Considering it is now November this information at the youngest is 11 months old, and at the oldest 23 months old (going back to January 1997). Um, I don't know about anyone else, but does anyone think that just maybe things have changed in this period of time???? I would hope, and I know I think, so.

<>

Sure seems to be a whole ot of leading going on here.

<>

Ok, I have no problem with this up to this point, except it would be nice to have some form of description as to what exactly failed each time.

<<. It was determined that this last failure would have caused a grid-wide failure had the plants been online.>>

This I have a problem with. Unless you are hooked up the grid you really have no way of knowing this. What could happen at a generation facility that would not be stopped be the built in safety precautions in the grid? I admit, I am no way an expert on the grid, but this statement just seems a little too bold to me.

Anyone else notice how Roleigh Martin seems to be the ONLY person getting anyone to talk off the record? The first thing I would like to inform Mr. Martin is that he needs to find another term. Off the record means just that, off the record. NOT TO BE PUBLISHED. If it can't be verified, which is a goodly portion of what off the record means, then it is not to be published. It does not allow for the reader to draw their own conclusions. We only have Martin's view with no independant way to verify.

Rick

-- Rick Tansun (ricktansun@hotmail.com), November 03, 1998.

I never will get the hang of the commands of this forum. Here is my post corrected:

"These paragraphs are from a utility insider who attended a state- wide session on Y2K and embedded systems in electric utilities in a Mid-West state in 1997."

Considering it is now November this information at the youngest is 11 months old, and at the oldest 23 months old(going back to January 1997). Um, I don't know about anyone else, but does anyone think that just maybe things have changed in this period of time???? I would hope, and I know I think, so.

"He reported that a leading electric utility engineering firm official and a leading Y2K expert with a leading electric utility industry group told the utilities the following:"

Sure seems to be a whole ot of leading going on here.

"In the testing of two coal-fire power plants (which were currently offline and being used as "hot spares") for year 2000 compliance, the clocks were simultaneously rolled over to the year 2000, causing immediate plant failure. In an attempt to better understand the failure, the roll over test was repeated. In the second test, the plants again failed, but a different embedded controller was determined to be at fault. The roll over test was repeated a third time in hopes of replicating one of the previous failures. In this test, the plants failed from yet a different embedded controller."

Ok, I have no problem with this up to this point, except it would be nice to have some form of description as to what exactly failed each time.

"It was determined that this last failure would have caused a grid-wide failure had the plants been online."

This I have a problem with. Unless you are hooked up the grid you really have no way of knowing this. What could happen at a generation facility that would not be stopped be the built in safety precautions in the grid? I admit, I am no way an expert on the grid, but this statement just seems a little too bold to me.

Anyone else notice how Roleigh Martin seems to be the ONLY person getting anyone to talk off the record? The first thing I would like to inform Mr. Martin is that he needs to find another term. Off the record means just that, off the record. NOT TO BE PUBLISHED. If it can't be verified, which is a goodly portion of what off the record means, then it is not to be published. It does not allow for the reader to draw their own conclusions. We only have Martin's view with no independant way to verify.

Rick

-- Rick Tansun (ricktansun@hotmail.com), November 03, 1998.

Rick, you say,

"This I have a problem with. Unless you are hooked up the grid you really have no way of knowing this. What could happen at a generation facility that would not be stopped be the built in safety precautions in the grid? I admit, I am no way an expert on the grid, but this statement just seems a little too bold to me."

Three issues with this paragraph.

(1) Built in safety precautions. Safety precautions are designed to prevent equipment damage or loss of life, or injury, not to prevent power loss. A good example is that of nuclear power: the plants are designed to safely shut down, resulting in loss of power, rather than risk a nuclear disaster. Since we're talking here about loss of power, safety precautions will actually enourage this to happen.

(2) According to posts from individuals who are involved in the design and sale of the actual generating equipment, that equipment is not the issue. The issue is whether or not power gets from the wires at the output of the generator to the point of use. This includes both the overall generation plant system and the distribution system. If the plant system fails.....the plant doesn't put out power.....then there's no way the distribution system can 'fail safe' in such a way as to deliver non-existent power.

(3) It's my opinion that the electric power industry knows, by means of simulations, exactly what it might require to take the grid down. We don't know this, but I really can't conceive of that data not being available someplace. While such simulations might not cover every possibility, they can be used as a sound basis for making such a statement. Is this the case? I don't know, but neither do you.

"Anyone else notice how Roleigh Martin seems to be the ONLY person getting anyone to talk off the record? The first thing I would like to inform Mr. Martin is that he needs to find another term. Off the record means just that, off the record. NOT TO BE PUBLISHED. If it can't be verified, which is a goodly portion of what off the record means, then it is not to be published. It does not allow for the reader to draw their own conclusions. We only have Martin's view with no independant way to verify."

Your understanding of the term "Off the Record" certainly doesn't agree with what I feel is generally accepted use. My understanding is that the person doing the talking is not to be tied to what's said in any way, such that the source is not jeporadized. His (her) anonymity is protected. The media uses such terms as 'a usually reliable unnamed source' to report off the record conversations. What the source says IS reported. Your attack on Martin on these grounds is unwarranted.

-- rocky (rknolls@hotmail.com), November 03, 1998.

Okay, I've read quite a bit of both sides on this utility issue.

Now most of us (if not ALL of us) want to hear potential impact. Does the above information mean it could take the grid down for a few days, a few weeks, a few months or is it down, period. The hows and whys are intriguing, but the bottom line is always the best. Even theorizing it is better than nothing. Thanks in advance for your thoughtful input.

-- (mass@delusions.com), November 03, 1998.

I found this posted on Nov. 2, 1998 on the Open Discussion area of http://www.euy2k.com

I suggest you read the whole posting and draw your own conclusions. -------------------------------------------------------------------- Northwest PUC notes asked in the Electric Utilities and Y2K Q&A Forum

Greetings all! I thought you might be interested in hearing what was said at the Public Utility Commission meeting in Salem last Wednesday. All of the big public utility phone/gas/electric companies representing the entire northwest were there reporting to the PUC their status on their y2k compliance. I took some notes as each speaker presented. I didn't get many names- unfortunately they went by too fast. I'll try to save any personal commentary to the end. In retrospect, I realize I didn't take very good notes. Unfortunately, much of the discussion was quite technical in nature and I found it was hard to understand let alone take notes on and figure out how it fit together. I primarily went as a concerned citizen to see what progress PGE (our local electrical utility) had made since their last mediocre report back in July, not as a reporter for the event.

Prior to the meeting overall the mood felt fairly light- some laughter, chatting, upbeat discussions among the suits. Lots of people in suits. I wasn't sure how many of the "public" were present short of myself. There was no press of any kind that I could see, no cameras, I appeared to be the only one taking notes. The meeting began right on time, very few stragglers arriving late.

My notes, I'll apologize at the start if some of them are unclear, people talk fast and I'm not a stenographer. I'll add any clarification in parenthesis if I can:

The 2 leaders of the PUC gave the reasons for the meeting, and stated that y2k is "still a critical issue."

The companies were requested to provide information as to the current status of their "business plan" and their "contingency plan" and then there was some questioning about what that meant. Business plan referred to "what they're doing now to remedy the situation", the contingency pan referred to "what they plan on doing if y2k affects something."

The gas companies went first:

The next group (the electrical utilities) began.

The NERC (North American Electrical Reliabilty Council) spokesman began the electrical discussion.

(I didn't catch his name but he was from a power comapany in Salt Lake City)

He said they send out monthly surveys to all of their power companies that NERC has contact with and gives quarterly reports.

He mentioned their website

(which I eventually found here it is: http://www.nerc.com/y2k/y2kplan.html)

The results of their most recent survey:

44 members responded

91% of all the companies have completed their inventory. 68% of all the companies are in the assessment phase. 28% are in the remediation phase.

He expected that all their members would meet their June 13, 1999 y2k deadline.

The PUC chair commented along the lines - You said 44 members responded, how many members are there?

There were 107.

There was some discussion about concern when the year rollover changes across the mountain/pacific time zone and "reducing loading"

(sorry no comment here on what "reducing loading" means, I just wrote it down)

They'll be doing some drills in April and September regarding the grid

There was some discussion about overgeneration and that all of the generating facilities would be up and running on Dec. 31, 1999 so there would be power on line to re-route if necessary.

Telecommunications was still considered a "critical" issue.

There was some positive discussion about having 16 hours prior notice if the far east has any problems. That is if the satellites are still operational... (general laughter in the room)

Nerc has a website, it took me a while to find it but here it is: http://www.nerc.com/y2k/y2kplan.html

(general laughter in the room)

Pacificorp spoke next.

Their computer systems were on track to be y2k ready by June 1999 deadline.

Regarding contingency plans they identified 1500 vendors that they deal with for operations. 130 were critical to their operations 60 had a time frame must be done by June of 1999. There were some stragglers that were not going to be ready

(my notes are a bit sketchy here, sorry- I wrote down the following but I don't remember how it fits together:

informations flow 100 different risk factors emmissions controls fall out of compliance)

There was more discussion about dependence on telecommunications and that Pacificorp's "system interacting with the internet" was critical

(whatever that meant)

The pacificorp rep then told the following story:

They invited Senator Bennett (Utah) to a test rollover at one of their generating plants.

(I think it was a geothermal generator, but I'm not sure)

He went on to say that (of course) they would test the rollover a few days before the senator was actually present to make sure that it really would roll over correctly.

He went on to say that when they did the roll over "2 of the 4 operating screens went blank" in the control room.

(I have a note that says "2 of 4 peaking unit" but I don't remember what that refers to)

It took them a while to get the problem with the screens repaired and suggested that one of the contingency plans for all of the utilities was to make sure that their parts "vendors are available on 1/1/2000"

PGE (Portland General Electric) was the next speaker.

The spokesman was an underling of Carol Lindenberg (PGE's y2k project manager)

He was very difficult to understand,speaking at a low volume.

(Here's everything I wrote down that I he said):

"we've only just begun" in assessment

reluctant to issue any dates

(The room was very quiet)

The PUC chair asked him if PGE had done any testing similar the the example from Pacificorp, he responded that PGE has done "no testing yet"

Bonneville was next up.

(and again my notes are sketchy)

The spokesman said they were 30% complete with remediation.

They had done contingency panning.

They spoke of their ability to do a blackstart plan quite easily, there was some lighthearted discussion that all it took to do a blackstart was to open one of the doors in the dams to get the water/generators moving.

There was some optimistic discussion about how on Dec. 31st that loads would be low, it was midnight, on a holiday, over a weekend.

(I have some more notes that again I don't understand. I feel like Dr. McCoy now- "Dammit Jim, I'm an audio guy, not an electrician!")

Underfrequency load shed Key interties to the east coast reducing allowable power so lines aren't overloaded.

The bright spot in the meeting came when Bonneville announced that they would not be affected by the phone company since their telecommunications is done on a BPA analog microwave system. And that sometimes it's better to be out of date on your equipment.

There was some discussion about other smaller companies related to BPA that have aggressive programs for y2k but are way behind...

The phone companies were up next

The mood for this meeting appeared to be pretty light, even at it's conclusion. I felt pretty confident that the utilities were o.k. Almost all of the presenters sounded like this y2k business shouldn't be a really big deal, and that they were on track for completion. Until I came home and re-read my notes regarding what was not said. I'm trying to not be a suspicious pessimist, but,

The spokesman for NERC gave these statistics:

44 companies responded (but didn't say that 63 companies did not respond and would not have had he not been prompted by the PUC)

Of those 44 91% have completed their inventory (8% have not & have the other 63 even started?)

Of those 44, 68% have completed their assessment (32% or 29 companies have not and have not even begun remediation. That makes 92 of 107 companies that have not completed their assessment phase-that's a lot of power companies in the Northwest.)

28% have completed their remediation-meaning 12 are finished remediation (72% or 95 of the 107 aren't finished with the remediation phase.)

None have completely tested their systems to make sure they won't have a problem.

That kind of makes me nervous. I think I did the math right. If I'm wrong please let me know. I guess I'm looking at this with a "the- glass-isn't- eve-half-empty" slant. But maybe we need to look at this that way. I mean PGE at least came out and said "we think we have a glass, but we dont know yet..."

The phone companies sounded better than I expected, though there seemed to be some holes in US west. Especially in their own IT systems.

I'll be especially attentive to the discussion about the "pipeline company" and who exactly that is.

Anyway, again my apologies for bad note taking. Here's my disclaimer: My notes on what I did have are accurate to the best of my abilities, and I have no greivances against any of the power companies that were represented. I'm just "Joe Citizen" trying to make sense out of this y2k business and not feeling like we (the people) are getting as much information as we can in this regard. If I didn't get some of the facts right, my mistake, if you've read this and have more questions I'm sure a phone call to your own local utilities will take care of any confusion you may have, if you can actually reach someone who knows anything...

thanks and keep in touch,

Jeff

Asked by Jeffrey D. Wold (jwold@cmdpdx.com) on November 02, 1998. ----------------------------------------------------------------------

-- Anna McKay Ginn (annaginn@aol.com), November 03, 1998.

I snipped some of the above posting to save space concerning the Gas and Phone companies. I inserted snip comments surrounded by < > and they didn't post. I apologize. Next time I will use another format.

Anna

-- Anna McKay Ginn (annaginn@aol.com), November 03, 1998.

Damn. It's not good - and looking worse.

Your statistics and notes were just fine - speaking to the original notetaker. Nothing critical was missed, just needs translation. As I figured - the grid and remote control (the cross-ties/frequency/microwave relays/telephone interfaces all are in that field, or measure things/report things in that field) are not "just fine".

"...informations flow 100 different risk factors emmissions controls fall out of compliance) ..."

That's why I figure the Congress needs to waive emmision control violations during some deined period, else these things will shut down rather thatn face fines under manual control.

-- Robert A. Cook, P.E. (Kennesaw, GA) (cook.r@csaatl.com), November 03, 1998.

Anne - in re your post - You can't assume that everyone who does not respond to a survey is doing nothing or is behind. GE for example is no longer responding to any Y2K surveys whatsoever - they were getting dozens of them each day. You can request Y2K information about a specific product and they will send it to you, but they aren't posting a moving target on the web either. Their Year 2000 (their preferred term BTW) project is, as you would expect, enormous and comprehensive since their products touch every part of power generation and control. And speaking of which (which is what brought GE to mind) the reference to the need for the need to interact with the Internet for control purposes probably refers to a transmission site controlled by GE's Cimplicity software and controllers which let you hook up over a network for control. As for the reference in the original post about a test in 1997 - what did you expect? You test to find out if you have a critical problem - then you work to fix it. I wonder what the pessimists will say when they start reporting plants have rolled over for testing after Y2K repairs and checked out OK.

Incidentally, about communications, the telco's are in a peculiar position right now. In a pretty recent 18 month period (forget the exact dates) usage of the lines went from mostly phone service and a small fraction Internet service - to the Internet breaking even with phone service - and now the Internet dominates communication lines world wide. This has led to massive equipment upgrades and replacements in just the last couple of years, and leads one to a curious speculation - that it is entirely possible that someone with Roadrunner or a T1 or ISDN line will have more reliable Internet service than phone service come 1/1/00. Curious thought no?

BTW, if anyone wants to read a lot of technical tedium about the new telco services and etc., ask and I will post some links. Be warned, it bores me, and I have to read this stuff to keep current.

-- Paul Davis (davisp1953@yahoo.com), November 03, 1998.

I was thinking the same thing as Paul. Because a survey is not returned does not necessarily lead to the conclusion that nothing is being done. I could see organizations that are waist deep into fixing things ignoring irrelevant surveys. When I am fairly busy, I ignore general wage surveys that come across my desk because I gain nothing from the completed product. So you can't automatically assume that the glass is half empty.

-- Slick (slick@hucheemama.com), November 03, 1998.

Rocky,

Thanks for the explanations on the bringing down of the grid. Like I said, I am far from an expert at the grid.

As for the use of "off the record", let's put it this way, in general journalism it is frowned upon at best. I have tried to pull it off a few times and been shot down by editors every time. It is sloppy writing if nothing else. I only tried it one I was first starting out (you know, when journalism was still that "exciting frontier" in my mind. Now it is just another job) It is not accepted by quite a bit of the industry.

In reference to Y2K it is even sloppier. One of the main things this forum has said time and time again is that every company needs independant third party verification on compliance, but we are willing to accept the word of one unnamed source if things are bad? (and please do not take any of this personally Rocky, I am just irked at the situation of unanmed sources, espically in relation to Y2K)

Rick

-- Rick Tansun (ricktansun@hotmail.com), November 03, 1998.

No, Paul and Slick, you don't just willy-nilly assume the glass is half empty. You apply this assumption when you recognize how vitally important the glass is. For example, with electric power, on which our entire civilization tends to depend on. Simply declaring "Gee, an unreturned survey must mean that they are just so busy working so hard getting it all fixed in time" just doesn't cut it.

-- Jack (jsprat@eld.net), November 03, 1998.

Rocky - thought of this after I posted and walked away fro a few moments. Generally, the use of off the record sources is supposed to work like this:

1. You get the hunch, and you start looking for whatever information you can find 2. When normal channels are not working you call up whatever contacts you may have, or can find, and ask them off the record about the situation. 3. Using the leads you get off of the unnamed source you then go and confront the correct people with the information you now have in hand (like saying "Well isn't it true that such and such?" "Well from what we have heard..." and so on) 4. If all else fails, there are no other ways to get the information, and it does not legally open you up to a slew of lawsuits, then SOMETIMES it is ok to use unanmed sources. Sadly though, a lot of reporters (read:major news networks and the such) now of days just skip phase 3 so that they can get on the air by dinner time.

What I said about no way to third party verify this Y2K information though, I do stand by.

Rick

-- Rick Tansun (ricktansun@hotmail.com), November 03, 1998.

Jack E. Boy,

Neither Paul nor I say that the glass is necessarily half-full, we both just stated that you can't automatically assume nothing is being done. I agree that this is a critical issue. But, do you see an IT manager wording 12 hour days for an electric company directing the work of others and working on gerenal administrative crap giving give two hoots about a survey that lands on his desk?

-- Slick (slick@hucheemama.com), November 03, 1998.

I think that We The People sure care about the progress -- or lack of it (more correct I'm sure) -- of Y2K fixes. Especially since all the 12-hour days of huffing-puffing DOES NOT mean that the problem is going to be fixed in time! "No news is good news" does NOT apply here, and quite frankly the lack of returned surveys should rightfully be viewed as a management problem.

-- Jack (jsprat@eld.net), November 03, 1998.

"Especially since all the 12-hour days of huffing-puffing DOES NOT mean that the problem is going to be fixed in time!"

And having an answered survey in yourhand based n today's progress does not gaurentee it will be fixed in time or it won't. It is a piece of paper based on things as they stand today.

And what exactly should motivate a company to reply to these surveys. If they are positive people scream that it has been through the PR ringer, that it is hyperbole and should not be believed. If it says we can't comment it will get the same reaction and if it says that things ar emoving slower than expected the Doomers will say "Aha! Run for the hills!" Surveys from indviduals or Y2K groups are a lose, lose, lose situation for anyone filling them out. In the case of the utilities, yes I am concerned, but I say just leave them alone. They are working on it and no number of surveys will make it go faster or easier.

Rick

-- Rick Tansun (ricktansun@hotmail.com), November 03, 1998.

Just for the record, I did not make any assumptions. The original poster, made the assumptions. I suggested that everyone read the post and draw their own conclusions. ;)

But now that you have brought the subject up....

We are not discussing just some silly old survey are we? We are talking about the Generals in charge of the Y2K war.

By the way Paul, I would love to read your information regarding the Telecoms. I worked for GTE for 21 years, 15 years in Special Services. One of my jobs was Administrator, Special Services-Data. I was the technical support to craft and first level management for all data communications.

-- Anna McKay Ginn (annaginn@aol.com), November 03, 1998.

OK Anna - here you go. The communications world seems to have changed more in the last 5 years than in all the time before. So much has changed that I think a telco engineer from the 70's would not believe the stuff that is going on now - even the language has changed. Its like riding a roller coaster trying to keep up - scary sometimes but a wild ride nontheless.

These are mostly from data.com - a pretty good trade magazine They involve the new phone system and what has changed

About halfway down the second page this link is some discussion of the traffic changes

http://www.data.com/issue/981007/world.html

this article talks about the local systems who are fighting the new net changes and deregulation

http://www.data.com/issue/981007/betrayed.html

this gives you links to pdf's that lay out the way the new stuff hooks up

http://www.data.com/issue/981007/roadmap.html

This is about some of the new equipment and how it is used

http://www.data.com/roundups/mux.html

This is the browse page

http://www.data.com/browse.html

You could also try www.nwfusion.com - you have to sign up for a password to get into their site tho so I won't give any specific links also - www.lantimes.com was good but they have quit publication - don't know where their archive has moved.

Network Magazine is also good - www.networkmagazine.com here is a pertinent article

http://www.networkmagazine.com/cgi-bin/newsstory.pl?98110302.htm

If you want any more just ask - these are the first ones that came to mind about the new systems.

-- Paul Davis (davisp1953@yahoo.com), November 04, 1998.

Thanks Paul,

I left in Dec. 1995. I am sure things have changed a bit. I generally get my information from the FCC web site. I appreciate other sources.

Anna

-- Anna McKay Ginn (annaginn@aol.com), November 04, 1998.