Emergency Declared at Hope Creek

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

Y2K is mentioned briefly here, as the reason an "emergency response data system" was not in use for this event (being upgraded for Y2K). This, too, is from Garden State Environet. I tried to fix the formatting but parts of it may still be hard to read ... - Judy

EMERGENCY DECLARED AT HOPE CREEK

Date: 990930

From: http://www.nrc.gov/NRR/DAILY/der.htm

Daily Events Report

U.S. Nuclear Regulatory Commission Operations Center

Power Reactor Event Number: 36244

FACILITY: HOPE CREEK REGION: 1

NOTIFICATION DATE: 09/29/1999 UNIT: [1] [] [] STATE: NJ

NOTIFICATION TIME: 22:22[EDT] RXTYPE: [1] GE-4

EVENT DATE: 09/29/1999

EVENT TIME: 21:33[EDT] NRC NOTIFIED BY: KEN BRESLIN LAST UPDATE DATE: 09/30/1999 HQ OPS OFFICER: LEIGH TROCINE ----------------------------- ------------------------------------------PERSON ORGANIZATION EMERGENCY CLASS: UNU DAVID SILK R1 10 CFR SECTION: LEDYARD (TAD) MARSH NRR AAEC 50.72 (a) (1) (I) EMERGENCY DECLARED JOSEPH GIITTER IRO STINEDURF FEMA FRANK CONGEL IRO ---------------------------------------------------------------------- UNIT SCRAM CODERX CRITINIT PWR INIT RX MODE CURR PWR CURR RX MODE ---------------------------------------------------------------------- 1 N Y 100 Power Operation 100 Power Operation ----------------------------------------------------------------------

EVENT TEXT

UNUSUAL EVENT DECLARED DUE TO A FREON LEAK FROM A CONTROL ROOM CHILLER INTO THE AUXILIARY BUILDING

The licensee declared the unusual event at 2133 on 09/29/99 due to a release of toxic gas (a Freon leak) deemed to be detrimental to safe operation of the plant. No radiological release is in progress. The 33' wind direction is from 146~ at 17.9 miles per hour. There were no protective actions recommended.

At 2117 hours on 09/29/99, the 'B' control room ventilation train chiller tripped resulting in an automatic start of the 'A' control room ventilation train. The trip was the direct result of a major freon leak dispersing into the auxiliary building. All non-essential personnel were evacuated from the auxiliary building. An Unusual Event was declared at 2133 hours on 09/29/99 in accordance with Emergency Classification Guidance Section 9.4.1.B, release of toxic gas deemed detrimental to safe operation of the plant. No personnel injuries have occurred, and all access to the auxiliary building is being controlled until follow-up actions are complete.

The 'A' control room vent train is currently in service. There is no indication of any freon in the control room, and the control room was not placed on recirculation.

The licensee plans to terminate the Unusual Event when the freon leak is isolated and when air levels in the auxiliary building are restored to normal.

The licensee stated that the emergency response data system (ERDS) has not been activated because ERDS is down for the performance of Y2K upgrades.

The licensee notified the NRC resident inspector, the states of Delaware and New Jersey, and the Lower Alloways Creek Township.

*** UPDATE AT 0204 ON 09/30/99 FROM KEN BRESLIN TO DOUG WEAVER

The leak was terminated because the full freon charge in the chiller dispersed (2,200 pounds), and the atmosphere in the auxiliary building has been restored to normal. As a result, the licensee terminated the Unusual Event at 0158 hours on 09/30/99.

The licensee notified the NRC resident inspector, the states of Delaware and New Jersey, and the Lower Alloways Creek Township. The NRC operations officer notified the R1DO (Silk), NRR EO (Marsh), IRO (Giitter), and FEMA (Caldwell)

Power Reactor Event Number: 36242 ---------------------------------------------------------------------- ---------------------------------------------------------------------- FACILITY: HOPE CREEK REGION: 1 NOTIFICATION DATE: 09/29/1999 UNIT: [1] [] [] STATE: NJ NOTIFICATION TIME: 18:58[EDT] RXTYPE: [1] GE-4 EVENT DATE: 09/29/1999 -----------------------------------------EVENT TIME: 11:33[EDT] NRC NOTIFIED BY: ART BREADY LAST UPDATE DATE: 09/29/1999 HQ OPS OFFICER: LEIGH TROCINE ----------------------------- ------------------------------------------PERSON ORGANIZATION EMERGENCY CLASS: N/A DAVID SILK R1 10 CFR SECTION: AARC 50.72(b)(1)(v) OTHER ASMT/COMM INOP ---------------------------------------------------------------------- UNIT SCRAM CODERX CRITINIT PWR INIT RX MODE CURR PWR CURR RX MODE ---------------------------------------------------------------------- 1 N Y 100 Power Operation 100 Power Operation

EVENT TEXT

LOSS OF CONTROL ROOM INDICATION OF RADIOACTIVE EFFLUENT RELEASE VIA THE SOUTH PLANT VENT RADIATION MONITORING SKID FOR GREATER THAN 8 HOURS

The following text is a portion of a facsimile received from the licensee:

"At 1049 [hours] on 09/29/99, the South Plant Vent (SPV) Radiation Monitoring Skid (RMS) was removed from service for a scheduled radiation protection surveillance. Following completion of this surveillance, the skid was returned to service at 1127 [hours]. At 1133 [hours], the SPV RMS skid began giving erratic indications including high effluent radiation, high particulate activity, and high noble gas activity. Radiological surveys of the plant vent stack and other associated areas revealed no change from normal radiological conditions. Checks of corroborating channels revealed no change from normal radiological conditions. Compensatory measures have been initiated [in accordance with] plant technical specifications. The effect of this failure is a loss of control room indication of radioactive effluent release via the [SPV] Stack. The loss of this assessment capability for greater than [8] hours requires a [1-hour] report to the NRC [in accordance with] the Hope Creek [Emergency Classification Guidance]. Maintenance technicians are currently troubleshooting the problem, and the cause of erratic indications is not yet known."

The licensee plans to notify the NRC resident inspector. [END QUOTE]

-- Anonymous, October 01, 1999

Answers

Most interesting - not so much the event itself, but the fact that the Emergency Response Data System (ERDS) *still* isn't upgraded at this facility. ERDS is a data communication interface between NRC and all of the U.S. nuclear plants to transmit plant data and information when the plant has an event that requires classification by the Event Classification Guide.

Let's flash back to 1997 for a moment, and transcripts from the NRC/Nuclear Energy Institute meeing of October 7, 1997:

John Jolicoeur of the Office for Analysis and Evaluation of Operational Data described the efforts to update the Emergency Response Data System (ERDS) for the Year 2000. His presentation included some background on the ERDS and the importance of date fields to the system. He then detailed the main features of the NRC's ERDS action plan and the current schedule. He is planning to complete testing of the NRC Operations Center ERDS in November 1998. He indicated that where licensees are planning to make software changes to their ERDS interfaces, he needs to know what upgrades licensee are performing on their ERDS software and the projected completion dates. This information is necessary so that he can adequately plan and schedule testing of the NRC-licensee interfaces.

This has been a known issue with actions required on both the part of the NRC and licensees since the 4th quarter, 1997. And here's one example of a licensee that, as of the 4th quarter, 1999 still hasn't performed the necessary upgrades. How many more of these type issues exist?

Note: ERDS is *not* essential to plant operation, but it's certainly one of those efficiency enhancers if an emergency exists that requires monitoring and input by NRC. The ERDS system was a post-TMI regulatory enhancement. How many non-safety related, non-mission critical "efficiency enhancers" can a plant afford to lose before the aggregation of these losses becomes problematic? I dunno. But each one decreases operational response capability and safety margins, regardless of whether or not the affected system is "safety related".

-- Anonymous, October 01, 1999


Rick,

Hope Creek hadn't scheduled their upgrade until this week, according to what they reported to the NRC when asked back in June. (www.nrc.gov) I don't know why they waited until this week, but it was their intent to do it now since June.

Doesn't make it right, but that was their plan.

TM

-- Anonymous, October 01, 1999


and just in time, a message from the oral office:

Clinton Orders Review Of U.S. Nuclear Facilities

http://dailynews.yahoo.com/h/nm/19991001/ts/nuclear_clinton_1.html

-- Anonymous, October 01, 1999


Monty - I think this order pertains only to fuel processing facilities, but I could be wrong. The article was not clear enough in that respect.

TM - I was aware of the schedule at Hope Creek - I'm just kind of (no, really) surprised that there are still unremediated nodes of ERDS still hanging out there. NRC "owns" ERDS - it's their system. While the ERDS platforms and software at NRC's offices in Rockville Md. may indeed be Y2k ready (not arguing semantics over ready and compliant), it seems to me that the overall system is only as "ready" as it's weakest link. Then again, I freely admit that I don't know that much about ERDS total system architecture...so maybe I'm wrong on this one too. ;-)

-- Anonymous, October 01, 1999


The NRC Y2K Audit done for Hope Creek on October 26-29, 1998, has a chart of the "Inventory of Mission Critical Digital Systems" which lists the Emergency Response Data System as having completed Phase II at the time.

The Audit document lists the six phases of this facility's Y2K project plan:

"The Identification (Inventory and Initial Assessment) Phase comprises the elements of awareness, inventory, categorization, and classification. The Detailed Assessment Phase comprises the elements of prioritization, analysis of initial assessment, vendor evaluations, utility owned or supported software evaluations, interface evaluations, and remediation planning. The Resolution Phase comprises the elements of remediation, verification, and contingency planning. The Validation Phase comprises the elements of Y2K validation testing, and contingency planning for systems failing Y2K tests. The Implementation Phase comprises the elements of planning and performing final implementation in the plant, initiating contingency plans and compensatory actions for failed or deferred acceptance tests, and decommissioning or retiring obsolete software and systems. The Closeout Phase comprises the activities that document completion of the Y2K activities for each system."

It appears that since a year ago (minus 24 days) the status of the ERDS system only progressed to Phase III (remediation) of the six phases until now, when upgrading for ERDS is in progress. The Auditors also stated, "The licensee has established a tightly controlled schedule for completing the Y2K readiness implementation phase for mission critical systems and components by September 30, 1999."

Since the "implementation phase" is listed on this document as Phase 5, after remediation (3) and validation testing (4), it would certainly appear that Hope Creek was not able to adhere to its September 30, 1999 completion schedule for mission critical systems, even if you envision some phase overlap or phase deletions. Also, as of July the NRC Chart of "Plants that are not Y2K Ready" which I'm assuming is what TM referenced, showed the scheduled date for completion of the ERDS system as 9/30/99 - not when their upgrade was scheduled to be done, but when it was supposed to be _finished_.

In the September 1999 status charts in the document at:

http://www.nrc.gov/NRC/NUREGS/SR1706/sr1706.pdf

the answer to why Hope Creek didn't have the ERDS system done yet is given as "The licensee is waiting for a replacement engineer workstation to be delivered." This is not to mention that other systems at Hope Creek have a projected completion date as of the end of this month. (ERDS is not the only system which missed the original Sept. 30 completion date for mission critical system readiness as given in the Oct. 1998 Audit.)

Like Rick, I also have things I wonder about. Since Hope Creek had assessed their ERDS back in Oct. 1998 as the Audit states (and that a year after there were known problems mentioned by the NRC), then either they did not put in a replacement order at that time, or it's taken the vendor almost a year to get the new workstation delivered, and even then it was not there in time to meet the completion schedule -- based on when it was supposed to be available.

Waiting on vendor delivery is given as a main reason for the lack of critical systems readiness for both nuclear and the bulk non-nuclear utilities with remaining "exceptions". How NERC or the NRC can assume readiness will be achieved for these facilities, and that there will be time to test and implement the upgrades, when this depends on the quality of work and timeliness of vendors outside of a utility's control, I don't know.

-- Anonymous, October 02, 1999



Bonnie,

As Malcolm Taylor has so graciously informed us, even their new SCADA system had major growing pains for some months. The best minds on the vendor side had to try, try again to get it working correctly, and without it the plant had to go manual. Just getting the latest and best new equipment/system is no guarantee it will perform correctly "right out of the box."

-- Anonymous, October 02, 1999


As indicated in the PRN to the NRC, Hope Creek was in the process of implimenting the ERDS upgrade when this event occurred. It appears that Hope Creek has indeed completed the upgrade of ERDS, per the October 1 status report available at www.nei.org.

Regards,

-- Anonymous, October 03, 1999


FactFinder: Is Hope creek one of the 95 'ready' plants, or one of the 8 "non-ready" plants? This might sound like a sarcastic question, but it really isn't meant to be. If it's one of the 95 "ready" plants, then obviously there are some huge holes in the system, regardless of any other reporting which has occurred.

-- Anonymous, October 03, 1999

Factfinder, I've gotta hand it to the NEI. They must have more up to date information, or get it quicker than the Y2K Management at PSE&G (Hope Creek owners) do. I called the PSE&G Y2K info number and after the standard "all following statements are made under the Y2K disclosure.." etc. etc., a nice lady named Linda determined that she didn't have the up-to-today info I was inquiring about, so I was put through to a member of their Y2K Team, Ms. Portia Herbert. (Both Linda and Ms. Herbert were very efficient and polite, by the way.)

Ms. Herbert initially told me that the ERDS was still outstanding at Hope Creek according to the most recent report she had (Sept.22). Then she called me back a little while later to say she had tried to get an update and that the ERDS was now completed. When I asked her what the date of completion was, she said she didn't know but it was "sometime between September 28 and today." Since we know from the NRC incident report that the ERDS was down for repairs late on the 29th, I guess they must have finished the repairs, put the ERDS back into operation and done the testing to see if the implementaion was successful, all on the 30th, then immediately called the NEI to say, "Done!" so that it could be put in the report which the NEI put out on the very next day. And they handled the freon leak incident problems and its aftermath on the same day, too. I just love that kind of quickness and efficiency.

-- Anonymous, October 04, 1999


Bonnie,

That information you got from Hope Creek sounds just like some of the things that Jane Garvey was saying about the FAA at the end of September last year. Ho, ho, ho! You know, it just occurred to me, maybe that is why they call themselves the *Hope* Creek site. Without a full measure of hope, they could just be another nuke up a creek.

-- Anonymous, October 04, 1999



Bonnie: "I guess they must have finished the repairs, put the ERDS back into operation and done the testing to see if the implementaion was successful, all on the 30th, then immediately called the NEI to say, "Done!" so that it could be put in the report which the NEI put out on the very next day. And they handled the freon leak incident problems and its aftermath on the same day, too. I just love that kind of quickness and efficiency."

I myself am curious as to the timing myself. The scenario you present above makes a good "target" (a sitting duck in fact), but it's not very realistic.

ERDS is more of a simple communications protocol than anything, grabbing selected data from the plant monitoring computer and sending it to the NRC on some frequency interval. The software is not elaborate, nor is the hardware. A far more likely scenario is that any software changes (if required) were done before the implimentation and installed on the workstation/server. The "same day" stuff is a strawman of yours, although it certainly IS possible to install and test on the same day, especially with ERDS.

As far as checking status, it is likely that NEI was the one following up prior to presenting the latest October 1 status.

If ERDS was completed on the 29th or 30th of September, or even October 1, then NEI provided the correct status for Hope Creek (Y2K Ready). If ERDS was completed after October 1, then NEI or the Hope Creek team screwed up on the status.

And someone at Hope Creek knows the date that ERDS implimentation was completed.

Regards,

-- Anonymous, October 05, 1999


Factfinder, first you said the scenario I presented to fit Hope Creek's implementation into the known time frame wasn't realistic. Then you also said that it "IS possible to install and test on the same day, especially with ERDS." Despite that contradiction, let's assume, as you stated, that ERDS is "more of a simple communications protocol than anything" and the "software is not elaborate, nor is the hardware". Yet the industry knew about the need for an ERDS upgrade two years ago, and Hope Creek knew this simple system that supposedly might only take a day to implement and test needed to be done a year ago, according to their own records.

So we have this simple potential one-day fix/upgrade to a system which was put in the "critical" classification by Hope Creek, taking a year to be implemented, despite all the pressure from the NRC to be ready by July 1. From my perspective, the fact that it took a year before this "simple protocol" upgrade was done in the first place argues either that the whole of the process wasn't so simple after all, or the vendor did not find it easy to accomplish their part, or Hope Creek was stupid not to have gotten such an easy thing done in plenty of time.

And I certainly agree that "somebody" knows the exact time the implementation was finished, but since Ms. Herbert said she had asked for an update between the first and second time we spoke, either she wasn't given that specific info or she wouldn't give it to me. Maybe that inconsistency doesn't make you wonder, but the only time in our two conversations that there was a big pause/hesitation from the lady was after I asked when ERDS was finished and then ended up being given that range of dates.

I could not help but be reminded of when my local bank announced their Y2K readiness on July 1, and then a few weeks later I saw their anouncement that the ATM's would be unavailable on such and such a time due to replacements being put in. After asking questions, it turned out that, well, they were "ready" because things like the ATM upgrades were "scheduled" even if the work hadn't actually been finished yet. Was my bank's type of viewpoint/rationalization a completely isolated case? From what I know of human nature, I'd place a bet on the side of its being quite widespread versus isolated. Could Hope Creek have adopted a similar mindset when informing the NEI of when they'd be finished? I'd certainly say that's a possiblity, given the date ambiguities.

It's called "fudging", or even "well-intentioned rationalization". It's not called truth.

-- Anonymous, October 07, 1999


FactFinder,

I really think you are being an apologist for Hope Creek on this. Do you see why we, outside looking in, are so suspicious of the work being done, let alone the quality of the work? Last minute rush, rush, to get on the "ready" list. I swear, if you had taken your car in for an engine repair, and day after day, week after week, itwasn't done, but you got lots of new excuses, you'd be irritated. And if you finally got fed up enough to tell them if it wasn't done by this Friday you would start a law suit, and then low and behold, just before closing on Friday you get the call, "it's done." Sure, you would feel real confident about the work, sure you would.

-- Anonymous, October 07, 1999


Moderation questions? read the FAQ