From the Dec. 14-17, 1998 NRC Audit at:

http://www.nrc.gov/NRC/Y2K/Audit/Y2K50382.html

Definitions: Mission Critical - An asset that, if lost and for which there is no timely remedy, would disable or otherwise adversely impact the ability of the Entergy system to: (a) operate in a safe manner, (b) provide service to customers, (c) generate revenue, or (d) avoid legal exposure.

Important - An asset is considered Important if it is not Mission Critical but places an unanticipated operational requirement on plant operators, causes some degree of regulatory impact, has moderate cost impact, or has a wide scope of usage.

Desirable - An asset is considered Desirable if it is not Mission Critical or Important.

Unless specifically noted otherwise, the licensee did not formally assess and remediate desirable items. Remediation of these items may be done as time and resources permit.

Mission critical and important items will be verified to be Y2K compliant or ready by the deadline date of June 30, 1999. Desirable items will not be verified to be Y2K compliant or ready by the deadline date.

"The licensee identified 8 mission critical and 35 important applications that are potentially affected by the Y2K problem. The licensee also identified 272 mission critical and 365 important embedded systems that are potentially affected by Y2K problems. Assessment/testing has been completed for nearly all of the mission critical software applications, embedded devices, and spare parts identified by the licensee. Assessment/testing has not been completed for the software applications, digital devices and spare components important to operations. Table 2 provides a list of mission critical software systems reviewed by the audit team. Table 3 lists the embedded devices that were reviewed by the audit team."

When I checked out Table 2 and Table 3, I saw that the Audit team had reviewed 13 Mission critical and 4 Important Software Applications. Under "Completed remediation" only 2 of the reviewed software apps were done, and none of the Important software apps were listed as completed.

In the Embedded systems that the auditors reviewed, there were 33 items listed as Mission critical, with 9 listed as "Completed remediation". There were also 7 Important items reviewed, with none listed as completed.

In the Validation and Certification section of the Audit report it states:

"Of the 275 Mission Critical embedded items in 25 work packages, 57 embedded items have been completed in 15 packages. Two packages were to be completed by December 31, 1998, and the remaining eight packages will be completed by June 1999."

In the Y2K Program Management section of the Audit it states:

"The licensee is preparing for a W3 refueling outage in February 1999. Consequently, the systems and operations staff, whose support is necessary for addressing Y2K issues at W3, are not available to review Y2K work packages at this time. Directed support from the W3 Y2K Project sponsor will continue to be necessary to expedite Y2K progress at W3. The licensee has committed to provide additional resources on the Y2K project to meet the established schedules."

Under Electric Grid Issues:

"The Global Positioning System (GPS) receiver at the Beaumont center is being replaced with a Y2K compliant receiver. The GPS receiver provides the standard time, which is used as the base time for monitoring grid frequency."

Under Critical Suppliers:

"The licensee has prepared a list of critical suppliers, and has compared their list to the corporate list of suppliers. Letters have been sent to critical suppliers, but W3 has gotten only a 10-25% response. EOI is following up on the responses. The licensee is planning on stockpiling a 30-day supply of critical supplies as a contingency."

-- Anonymous, March 08, 1999

Answers

Bonnie:

I am very interested in your assessment of this information. Here are some of my thoughts as a software developer:

1. "(a) operate in a safe manner, (b) provide service to customers, (c) generate revenue, or (d) avoid legal exposure..."
   
   This is where I think the bulk of the problems reside with this assessment. The standard is not quantified. What is "safe manner" or "provide service?" These are things anyone could argue over. A system specification is supposed to be quantifiable and measurable. For example, it may be a good idea to specify that "the system is considered a saftey risk if it is thought to have a 50% chance or greater of directly causing loss of power to a hospital..." or "the system is considered a safety risk if it creates exposure to radiation above xxx% on within such and such a distance..." or "the system is considered a safety risk if it says 'safety' somewhere in the documentation of the system..." or many, many, others.
   
   The same goes for "provides service to customers." Does this mean "any system that provides electricity to customers...", or "any system that interfaces with the customer directly..." or "any system that interfaces with the customer directly, or indirectly through at most a single person or computer system or control..."
   
   As a customer, I am not so concerned about "legal exposure..." and I feel a bit alarmed that it would be listed with "operate in a safe manner.." as a reason for being critical.
   
   
2. "Assessment/testing has been completed for nearly all of the mission critical software applications..."
   
   "nearly all" of the assessment... means to me that "we cannot realistically estimate the amount of time required to repair our mission critical systems because there are a few systems for which time estimates would be arbitrary." I would suggest that assessment be prerequisite to a "remediation plan" audit.  It wouldn't be a bad idea to audit the assessment phase and planning phase separately in my opinion.
   
3. "'The Global Positioning System (GPS) receiver at the Beaumont center is being replaced with a Y2K compliant receiver. The GPS receiver provides the standard time, which is used as the base time for monitoring grid frequency.'"
   
   I find this very interesting.... This makes me think that there is increased risk that the GPS rollover may have an affect on transmission of power.  It depends on why the GPS receiver was replaced...
   
4. "Letters have been sent to critical suppliers, but W3 has gotten only a 10-25% response."
   
   This is proof to me that they are late.  If the suppliers are truly "critical", then what does it matter if they are getting their own safety systems fixed?   If one of their suppliers causes them to have a safety risk, then workers or customers are at risk.

-- Anonymous, March 09, 1999

Reporter, it would take more time than I have right now (and more space here than is practicable) to go through a complete assessment of this Audit, and you and Bill Watt (on the first Audit thread) have already mentioned several concerns. So I'll just relate what I found most disturbing about the Waterford situation, and that involves what was NOT mentioned in the audit as well what was.

The Auditors wrote:

"the initially established deadline for the Mission Critical assets will not be met at W3, and this may impact the deadline for completing the Important asset work projects. Additionally, there does not appear to be sufficient flexibility in the schedules to account for unforseen delays in completing the work projects."

[My translation: There's been slippage in the original estimates for completion of the project and Waterford is already behind schedule. There is absolutely no leeway left for any further delays. Their backs are to the wall.]

"To address the above schedule conflict, the licensee established a revised schedule for Y2K readiness of mission critical and important system projects by June 30, 1999. The audit team believes that, with the additional resources the licensee plans to obtain in the immediate future, the above Y2K readiness schedule appears to be achievable."

[ In order for the Audit team to be able to say there was even a chance that Waterford might finish in time, Waterford had to make a new schedule for progress and promise to put more people to work on it to make up for already being behind.]

Assessment/testing has been completed for nearly all of the mission critical software applications, embedded devices, and spare parts identified by the licensee. Assessment/testing has not been completed for the software applications, digital devices and spare components important to operations.

[This Audit ended on Dec.17, 1998, one week before Christmas. The detailed assessment was not finished either for Mission Critical components or for Important components. "Nearly all" does not = Done. They were scheduled to have the detailed assessment for both Critical and Important items done by Dec. 31.]

"The licensee was preparing for a refueling outage scheduled in February 1999 at the time of the NRC audit. Consequently, the systems and operations staff and particularly the mid-level management, whose support is necessary for addressing Y2K issues at W3, were not as available as at other plants."

["Preparing" for a refueling outage scheduled to begin 6 to 8 weeks after the Audit kept Y2K staff from being available to the Auditors. What does this say about the priority of Y2K remediation at Waterford? The impression is also given that there is not enough staff to do both Year 2000 remediation and their regular jobs. This implication does not bode well for being able to maintain the "revised" schedule either.]

"The licensee was scheduled to complete its detailed assessment and certification of mission critical items by December 31, 1998. This schedule has not been met because the licensee is installing new equipment during the in February 1999 refueling outage, and testing and remediation cannot be completed without this equipment."

[This statement is a real doozy. Just reading the last sentence makes someone think, OK, they have to put in new equipment and of course they can't test it until it's installed. However, the "testing and remediation cannot be completed" is rather of a red herring since these statements were made under the "Detailed Assessment" section of the Audit report. Paying attention to the section of the report where these statements are found, and putting both sentences together, it's clear that it's the *Assessment* which cannot be finished until after the refueling outage, with remediation following after. Notice the report says, "testing and remediation cannot be completed without this equipment", NOT remediation with testing afterward. The "testing" before remediation the Auditors are referring to is the Assessment testing. How in heavens name anyone can figure that Waterford can complete their detailed assessment, and the following remediation and testing AFTER the February outage and before June 30, I don't know. Frankly, from the tone of the report, I don't think the Auditors think it can be done either.]

What is conspicuously absent from the report is any mention that since work will still be in progress AFTER the outage, what happens if they discover something that needs to be replaced and/or tested when the plant is down? To my knowledge, they certainly wouldn't be scheduled for another outage in 1999, after the February one. Also, since the Auditors stated that the additional resources promised by Waterford would be needed for the revised schedule to be successful, and that there was insufficient "flexibility" to withstand any further delays, it's a puzzle why there will be no followup Audits of Waterford to check on them, or to check on other Entergy units which might be in the same situation.

On the 10-25% response from critical suppliers -- what? They don't know whether it's a 10% response or a 25% response, or something in between? So much for their contingency planning. It's also not stated what those responses they did receive said. The Auditors also suggested that more staff is needed in the contingency planning area than the one part time person assigned to lead that project. Umm hmm, talk about obvious suggestions.

To sum up, I personally found the Waterford Audit very depressing reading.

-- Anonymous, March 11, 1999