Report from the trenches

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Two weeks ago I posted a c.s.y2k report from "Mickey" who works for a large power generation utility. This was a 'good news' report: he basically felt that his own company was in good shape.

Since that time he's attended a conference in Boston. This is his report of that conference. This is the 'bad news' report. I have edited the post to delete the name of a specific utility.

_________________________________________________________

Hi All,

Well, I'm back from beantown. We had a lot more people show up than anyone expected. We had representatives from 18 PGs (all either medium or large in size). These were either employees or the PGs themselves, employees of third parties doing the remediation (i.e. IBM, EDS, D&T...), and consultants working on these projects. All attendees were in the technical end of the spectrum (no PHMs). Friday night was devoted to handshaking and getting-to-know-you chatter. The real fun began on saturday.

The topics on the agenda were:

1. Code remediation methods 2. Data aging 3. Plants remediation and embedded processors 4. Contingency issues

I really wish I could say that this event left me all warm and fuzzy. It didn't. Once again I was left with the feeling that the major PGs don't really care if they can generate and deliver power, if they can't bill for it. As to billing for it, a large percentage of these companies use Arthur Andersen's CSS or Customer/1 (same fish, different newspaper). This package is purported to be compliant as delivered. All users add they're own mods, and therefore feel the need to test for compliance (as if the vendors word would have been enough!!). As we are currently testing the same system at SoCo, it was of particular interest to me. The type of thing I heard made me pray that they hadn't modified it past the point of compliancy. Here's a little example of the kind of things I heard:

XYZ Company has certified their CSS system as compliant. When I asked about their aging strategy, I received an e-mail consisting of an Excel workbook. CSS if rife with dates. Most of these are DB2 dates, and as such, present no problem in aging. There are, however, 182 derivative dates. These are partial dates, defined to the database as Character(4). They contain dates in formats such as YYMM, MMYY, MMDD, etc. The problem with aging these dates is that if you wish to age a date such as 0998 (MMYY) 45 days, what month do you arrive at. If one assumes (there's that word again) that the "DAY" portion of this date was a "01", then one arrives at 1098. If, however, this date was a partial date, derived from a DB2 dates such as '1998-08-17', and was aged 45 days, then the correct derivative is 1198. Just to cloud the issue (as if it needed clouding), some of the base dates from which the partials are derived have sentinel values (0001-01-01, 9999-12-31). In reviewing the actual production data, I found that some of the derived dates had valid values, even though the base DB2 date had a sentinel value. I asked Com-Ed how they handle this anomoly. Their response was that they eliminated these type of records from the testing scheme. What the did was to dummy up some data, arbitrarily decide which fields to age (based on whether they could determine its origin), and called it compliant. The really frightening part is that they spent 50 days going through this fudged aging. I heard similar stories from most of the other PGs. If this is an example of their testing and aging strategy, they are dead men walking.

Next. Plant remediation. After lots of chatter and bs, the net of this was that lots of these nimrods are actually in a fix-on-failure mode. Oh, they are doing testing all right. Just not the kind that has any value, unless the goal is to be able to say that "Well, we tested it and it worked fine at the time". In other words, their goal is to avoid litigation, not to ensure the flow of power. Now, I am not going to paint all with the same broad brush. Suffice it to say I heard enough to make me sure that the national grid won't be there after 2000-01-01. To those PGs that were taking the matter of plant remediation seriously, I asked the obvious question, which brings to the last topic,

Remediation. I inquired to all as to the nature of their contingency plans. The general response was one of "In contingency for what?". Even those PGs that had plans, these plans were of the nature of what to do internally should one of their plants fail. Not once did I hear mention of plans to take their generator off the grid and maintain the supply of power to their customers. I also didn't hear of any plans to refire their nuke should they be forced to shut down. The problem here, as has been discussed before, is that it takes a tremendous amount of power to fire up a cold core. Smart money would do something like running a direct wire from a hydro or fossil plant that is known to be compliant to the nearby nuke. This would allow the full output of the plant to be used to restart the nuke. In this scenario, it is possible to restart a cold core in under 24 hours (we have proved this). When I mentioned this to my contemporaries (term used loosely), I was met with lots of confused looks, as if to say "Now why would one go and do that".

The short of it all is that, imho, if you don't live in the southwest, it is likely to be a dark New Years Day.

Wish it were better news, but it ain't

mickey

-- rocky knolls (rknolls@hotmail.com), September 05, 1998

Answers

PS -- Paul: You might want to track down the original posting -- and then buy a generator.

rocky

-- rocky knolls (rknolls@hotmail.com), September 05, 1998.


Educate me. I've worked nukes before, it would appear that grid failure = almost immediate generator trip = scram, startup standby diesel generator(s) begin recovery with on-site power to the point of reconnecting to grid. There is where is see it getting dicey, since central control and synchronizioing to the grid/coordinating off-site substations can't be readily done.

I've not seen coal/hydro/oil plants with reliable backup power on site, nor the people who have as well trained in emergency procedures, not with as many backup systems. Stand-by gas turbines appear most capable of self-lighting, but they are relatively scarce nationally and are only rated for surge power capability, not baseline power. So, if a direct cable could be installed (free of grid?, you certainly couldn't fabricate, lay it in and route it in less than 1 year) wouln't the power end up from the nuke to the coal plant?

-- Robert A. Cook, P.E. (cook.r@csaatl.com), September 05, 1998.


I'm only guessing from reading this guy's posts. I believe that the particular company plans to have to cold start their nuke, probably because they anticipate a regulatory shutdown prior to 2000-01-01. They had decided that a dedicated fossil fuel plant (determined to be compliant)would be used to provide restart power.

They also anticipate that they may have to disconnect from the grid, ergo the direct line to the nuke from the smallish fossil plant.

Not a nuke engineer, so this is what I interpret. Make any sense?

-- De Lewis (delewis@inetone.net), September 05, 1998.


Doesn't make sense logically, maybe to lawyer though. You can test "everything" first by isolating all controls to rod control fuses (or equal) then testing even on-line items. The simulators and off-site training facilities mock up the remainder adaquately.

Startup from cold plant conditions are the hardest most complex evolutions to do, and plant operators start up rarely from cold iron. = So I'd never recommed trying to strart from cold iron under extreme conditions. If everything is running on midnght - 1 hour, keep it that way. It's possible too, that a full scram won't be needed, remember, we really don't know how the grid will perform under this type of "lack of control + bad control + bad communications + bad programming and imbedded chips" problem.

The nukes are addressing the problem, probably earlier and more seriously that any other system. They have a long history of good QA and over-engineering, and work from a small group of venders used to high QA needs and testing. Everybody in the industry takes their jobs seriously. So my experience tells me that the palnts themselvees will likely be safely operated, but I can't garantee you power coming out. If they are shutdown due to regulatory fear of "spooks", then i can assure you there will be no power coming from anybody else.

-- Robert A. Cook, P.E. (cook.r@csaatl.com), September 05, 1998.


Besides the southwest, wouldn't hawaii be a relatively safe (read "independent of the national power grid) place to be?

-- saranealy (keithn@ptd.net), September 05, 1998.


Moderation questions? read the FAQ