I agree completely with Factfinder's statements on the "UK Nuke alert regarding 9/9/99" thread, when he said he is "a firm believer in diverse redundancy for critical systems" and that "I believe that this is a topic worthy of discussion in it's own thread."

With that in mind, I came across the transcript of an ACRS (Advisory Committee on Reactor Safeguards) meeting held May 4, 1999. The topic was, "Safety Research Program". Parts of this transcript gave me the willies, because it seems that there is now an ongoing research project and part of that involves Virginia Power's decision to go to a digital Reactor Protection System instead of their present analog ones.

Dr. Miller: "Your Virginia Power application, is that going to be only the reactor protection system, or is that going to include other systems beyond the RPS?"

Dr. Johnston: "It'll just be the reactor protection system. As I mentioned earlier, they have -- Virginia Power has been, you know, very anxious, very interested in working with us. They have given us all of the detailed schematics for the current system, and what we've done there is we sort of had to reverse engineer, to say okay, here are the schematics, you know, what's the function that it's actually performing?

What we're working on now is trying to get from them some of the documentation, a requirements document, specifications document. It turns out that those things are proprietary to Westinghouse and they're working through the mechanisms of trying to get that information released to us. But it would be the reactor protection system."

DR. MILLER: "That's currently all analog?"

DR. JOHNSON: "It's currently all analog, and what they're looking at now is the replacement of their systems with digital systems. It's my understanding that they are undergoing a retrofit process now that will do Surry first and then following that I think is the North Anna facility."

The entire meeting is an information gathering session about incorporating digital I&C (instrumentation and controls) into nuclear systems, and the system reliability and modeling work which needs to be done. Dr. Johnson is a professor of electrical engineering at the University of Virginia in the EE department. He's a co-director and founder of the Center for Safety-Critical Systems. He's an IEEE fellow. More simply put, he's a researcher into modeling software and hardware reliability.

This is a long transcript but deserves a very careful reading. It is only related to Y2K in that the Year 2000 has put great focus on digital system reliability, and as Factfinder wisely indicated, looking to possible future impacts and how redundant systems should be handled does seem to be worthy of serious consideration.

Some highlights include the NRC funding difficulties for research into digital reliability and the difficulties of assessing relibility:

Dr. Calvert: ".....say this is the reactor protection system. We want to determine if there are any errors that happen from hardware or software that would -- "

DR. POWERS: "If you can determine it at that point, how come the engineers that developed it couldn't determine it?"

DR. CALVERT: "That's a good question. It seems as though engineers want to design for function and not for assessment. They don't like to assess it very much. They just want to make it. And Dr. Johnson has some good examples of this in his research, is that the -- and from my experience in inspection is that the intensity of trying to make a function work with hardware and software is the entire focus. It takes almost everybody to do that. And there's nothing left over for saying what happens if this fails from one reason or another. There are various people who do quantitative failure mode and effects analysis, but the ones that I've seen are very superficial. Some of these things require more of a detailed knowledge of the equipment and the software than they probably know about."

This is later summed up by saying that "making something work is a full-time job" and "finding a fault is a full-time job. And the two just don't match each other."

What do you readers think about the trend toward more digital controls? Is a top-notch hardware-software safety reliability model enough? Should analog systems remain? What about backup system diversity? Dr. Johnson's research deals with safety critical industries -- petrochem, aerospace, as well as nuclear. Will our critical industries benefit from further digitalization? They must think so to be investigating and planning for it. Should the public be aware and/or have input into the future of critical industry safety? Has Y2K had a good impact in focusing awareness on digital systems' reliability?

What are your thoughts on any or all of these questions?

-- Anonymous, September 10, 1999

Answers

Got so caught up in the transcript I forgot to post the URL!

http://www.nrc.gov/ACRS/rrs1/Trans_Let/index_top/ACRS_sub_tran/Safety_Res/sr990504

-- Anonymous, September 10, 1999

If y2k hasn't taught us the importance of redundancy for all of our critical systems, then shame on us.

And yes, the public should participate in informed discussions of policies that will impact the lives of generations to come. It's a fine ideal, nurtured on these boards by contributions like those Bonnie unfailingly makes. Thank you, Bonnie.

-- Anonymous, September 10, 1999

Bonnie,

This is a classic case of engineering short-sighted thinking. As Dr. Calvert states, in part: "It seems as though engineers want to design for function and not for assessment. They don't like to assess it very much. They just want to make it." This is almost exactly the problem that has arisen in all the newest models of commercial passenger jet aircraft. There should be at least basic analog instrument backups for ALL the critical readouts, in case the CRT displays go down, as they sometimes do. Or, we should be told in clear terms just why this is not needed, and it should be debated. If it were openly debated, I have no doubt that the analogs would remain.

-- Anonymous, September 10, 1999

FactFinder,

That was a superb posting. You have moved right over to the Malcolm Taylor side of the table with this one. I congratulate you! You are right-on in spotlighting this whole trend toward digital control, and the inherent weakness if that is all there is to rely on. I would like to offer a few thoughts about this from the perspective of airline flying.

First off, I am not opposed to digital control, per se. As you have noted, it is extraordinarily precise and predictable, when it is operating the way it was designed to do. In supersonic flying, for sure, the computer driven aircraft is the only way to go. Humans just can not react quickly enough to attitude changes at 2-3 times the speed of sound. The tendency of humans to overcontrol excursions has to be dampened by computer curves. Literally, the computer takes the command signal, input by the pilot, and determines how much, or how quickly, it will allow the actual change to procede. Computers can, and do, make it impossible to exceed a predetermined bank angle, or pitch angle, no matter what the pilot may be doing with the control yoke. No loops or barrel rolls possible here, like in the "good old days."

In the Boeing 747 models that I was flying we had at least 2, sometimes 3, autopilots. These would be used singly for normal cruise flying, but for a coupled-approach to an auto-land situation in bad visability, such as fog, we would use all of them together. They "talked" to one another, compared values, selected a mean average, and even ignored bad commands from any single system that "didn't fit the curve" that was established. Nevertheless, I have seen cases where the coupled approach got real squirrely, with large excursions from normal, late in the approach. We would routinely do these coupled approaches in good visability, good weather, to verify the integrity of the systems, and report on that, so that future crews, or maintenance, could get a picture of the actual equipment in that airplane. The fact that they could do some weird things, unexpectedly, meant that it was a "trust, but verify" situation for me, when they were actually being used in real weather conditions. I was "all eyes" so to speak, ready to disconnect them and hand fly it, either to complete the landing or do a missed approach and go-around.

There is usually pretty good indication that the automatic system is causing some anomaly. Perhaps a conflicting set of commands being carried out, such as one control trying to pull up while another control is trying to move the pitch trim in the opposite direction. This is good warning that it should be disconnected before it trips off and leaves you in a very precarious and unstable situation. These things were rare, very rare, but always lurking there to catch the complacent or unwary, or for that matter the very tired.

Another area is generator control. We had 4, 30kva generators, on some models, pumping out 120vac @ 400cps. Usually these were kept synchronized together to supply a "grid" for the system. Occasionally, a generator would not want to synch with the rest and could either be shut down, or "islanded" to supply only its own bus. If islanding was chosen, it was to gain the use of that electric power, but required closer watch on loads and be ready to shut it down and supply its bus from the other systems. Digital control does not offer this option, only manual control, yet it was a good thing to have available if there was still a long way to go. We certainly weren't going to land due to the loss of just one generator.

Pilots were never part of the design loop in aviation. Line pilots, that is. We got what they made, like it or not. We didn't even have input on the cockpit seating, where we were going to be spending more than 1,000 hours per year. The designers and engineers just gave us the seat they thought was OK, for the most part. Sure, occasionally some chief pilot of some major airline got a special seat built for their airline, but again, not with the input of pilots in general. That airline got the seat the chief pilot liked, period.

Lastly, we have heard about the digital takeover of railroads, right down to the switching. Look what has happened. All the old manual, or mechanically controlled, switches (or points, as they call them) are gone. Everything is now computer controlled. I read that these can be manually moved, by a line crew that stops at each one and physically shifts it, but the slow down of freight movement is dramatic. So, to comment on your post, I would say that digital is great, when it is working correctly, but to remove all other manual backups for economic reasons alone is folly. Safety is certainly compromised in a totally digital environment. It comes down to a question of what safety is actually worth, dollarwise, in terms of human life. I think we have been given an answer sometimes. No much. Let them sue.

-- Anonymous, September 11, 1999

Let me make a clarification on the generator comment above. I was not referring to the 747 for that generator situation. On the Boeing 747, there were much larger generators, capable of 54kw continuous each, with a 5 minute allowance of as much as 81kw each. Cooling was the critical factor in any overload scenario. Technically these were 60kw capable continuous, 90kw surge, but were derated by 10% on each limit for service reasons.

-- Anonymous, September 11, 1999

Gordon, I found your remarks about the use of redundant autopilot systems very interesting, I have no experience with aircraft systems, and have read about the subject only lightly. These systems talked to each other as you described, and that sounds similar to a few other systems I am familiar with (but not to a great degree) in the electric industry. I think that the design of the communications between redundant systems could be a critical design point, with a potential for common mode failure if not implimented properly, ie, also redundant, PLUS any "decision" logic would have to be error-free. In the example you gave, somehow I don't think that criteria was met, and I also don't think it is always met by a lot of redundant systems. I will be examining such systems in the near future, and I will go over them very closely.

I should point out that I do indeed see Y2K as a potential "common mode" failure in digital systems, primarily in higher level control systems (I know, surprise, surprise, but I thought I should let you know I concur).

Regards,

-- Anonymous, September 14, 1999