I saw this particular incident posted on Gary North's website earlier today. It reminded me that we had discussed this same incident back in May. So, I went back and searched the archives to find the discussion, and am pasting it here for some "contrast", if you will. I've read no followup since the time that this problem was initially discovered (if one of our nuke friends has access to an LER database, it would be wonderful to read the LER on this particular incident). Here's the discussion from back in May:

---

This is from the April 27 Headquarters Daily Report at the NRC web site at:

http://www.nrc.gov/NRC/DAILY/99027mr.htm

You'll have to scroll down the page a ways to find it. What puzzles me is why they were installing an upgrade in the first place if a date setback is an acceptable solution? I can only assume that the upgrade was considered a more optimal fix than the date setback. Thoughts, anyone?

REGION IV MORNING REPORT

APRIL 27, 1999

Licensee/Facility: Entergy Operations, Inc., Arkansas Nuclear 1 2, Russelville, Arkansas Dockets: 50-313,50-368

PWR/B&W-L-LP,PWR/CE

Subject: Y2K TESTING OF THE STATION BLACKOUT (SBO) DIESEL GENERATOR

Discussion:

On April 23, 1999, Arkansas Nuclear One (ANO) installed upgraded software and a new operating system as part of their program to make the SBO diesel generator Y2K compliant. The SBO diesel generator has an advanced touch screen control/display system with control/display locations in both the control room and locally at the machine. The software and operating system upgrade was planned to be completed just prior to the regularly scheduled surveillance test, which was revised to allow verification of the functionality of the newly installed software. The test was initially performed with the computer date at the present date and then reperformed with the date advancing from the year 1999 to 2000. During the performance of both tests (i.e., with the computer at the present date and at the advanced date), the diesel would only carry a load of approximately 500 Kw, instead of its expected load of 4400 Kw. After verifying that the diesel would not properly load, the licensee removed the new software and operating system and reinstalled the old version. The test was reperformed and the diesel loaded as expected. To ensure that the diesel would remain operable for Y2K, the computer date was reset to 1987. The licensee has compared the old and new versions of the software in an attempt to identify the cause of the anomaly. No differences have been identified to date, and as a result, the root cause of this problem has not yet been identified. The licensee will continue their troubleshooting efforts in an attempt to identify the problem. The licensee has stated that no similar problems with newly installed software have been encountered. Furthermore, they believe that this anomaly is not a Y2K problem since the failure to properly load did not specifically occur when the date on the computer rolled over from 1999 to 2000. This information has been coordinated with the Office of Public Affairs.

-- Bonnie Camp (bonniec@mail.odyssey.net), May 20, 1999

Answers

This information has been coordinated with the Office of Public Affairs.

Boy, they put this one on extended spin cycle. :)

-- Dan Webster (cantsp@m.me), May 20, 1999.

---

As with the Peach Bottom event, this clearly proves out that testing is ongoing. And that's a good thing.

But, what struck me is that this is another instance of a Safety Related System (and you betcha that EDG's are Q listed items in the nuclear industry) impacted by Y2k. What is missing from this report is what would have happened had no modifications at all been made to the system.

It appears to me they were trying to do it the right way - installing a Y2k compliant software upgrade to the control system. But there was something in the new code (probably a non-Y2k bug) that prevented the gen set from properly loading.

My question to the nuclear folks: wouldn't setting the date back to 1987 (and I don't understand this one...it should be 1972 for calendar matching / encapsulation purposes) be considered a T-mod and require a 10CFR50.59 analysis? And I don't know site specifics at ANO, but I would also expect that they have more than one EDG.

-- Rick Cowles (rcowles.remove@waterw.com), May 21, 1999.

---

Rick, I too would presume that an operating system and software upgrade on an EDG would need be a modification and require a 50.59, in addition to possibly being in the SQA program. However I believe the chances of a 50.59 analysis catching a software bug are zero, since it would rely on someone elses (likely the vendors in this case) software testing and documentation.

Not sure why 1987 was used on the rollback, but it appears that dates aren't critical enough with this EDG. Hey, what about those guys who claim that logging is safety related ?? ;) Actually, logging and date stamps are sometimes important for meeting documenation QA program requirements, but there is always the mighty pen....

Back to the date thing, it may not be possible to roll the date back to 1972 with this system, depending on the computer and operating system used. For reference, IBM compatible PCs running DOS/WIN operating systems cannot be rolled back any further than 1980. Lets HOPE this SBO DG control computer isn't running a Microsoft operating system though (now there's a scary thought!)

Your analysis and conclusions of this event seem right on target to me, with the information that we currently have on this.

Regards,

-- FactFinder (FactFinder@bzn.com), May 22, 1999.

-- Anonymous, July 30, 1999

Answers

Most of my worst "doomer" fears seem to be comming true these days. I'm seeing industries wake up slowly to the fact that they have REAL problems and the spin becomming less prevalent. We never did explain the reason for the deletion of the report on digital nuke failures posted earlier this week.

Now I read your post today rick along with a file I downloaded from the Nerc FTP site link so helpfully provided by Bonnie...

This is the file you want: 05/12/99 09:49AM 46,592 1-Disclosure-Issues-Swanson.ppt

Here's where you get it: ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/

Seems a bit blatant in regard to spin don't you think?

The tricky part! Arghhhh!

-- Anonymous, July 30, 1999

I did a search, but could not find the ANO SBO LER. I don't have access to a pure LER database, so it's not conclusive, but I doubt that this incident was reportable, since it is unlikely that the SBO DG had been returned to service and was apparently undergoing testing with the new software prior to returning to service. Problems indentified and fixed prior to return to service are normally not reportable, unless there are other circumstances that would meet the reporting critiera, such as same problem on an operable DG, part/software problem meets part 21 reporting criteria, etc.

I did find a few other interesting y2k incidents, including more details of one we have discussed...more later.

Regards,

-- Anonymous, August 02, 1999