Engineer and FactFinder have stated that some of the Y2K problems they have found only affect time-stamp logging. Are these logs used for anything or are they typically thrown away? Do any mainframes store the log data? Is the log data used by any computers at all? Is the log data hand-entered into any system by anyone?

-- Anonymous, June 04, 1999

Answers

Hi reporter. Here's one of many examples of how those time stamped records might be used...

One dark and stormy night, a car hits a pole, which causes the pole to fall and the conductors hit the ground, causing a fault. Within milliseconds, protective relays at both ends of the line see the fault and tell the circuit breakers to trip, thus isolating the problem. All the customers see is a quick flicker of their lights.

Those relays can then send data back to the operations center, telling the operators what kind of fault, and a calculated location of the fault. This helps them find the problem quickly (if some customer hasn't already called in). This data is time stamped by the energy management system at the time the information is received, and the relay does NOT send any date-dependent data.

The next morning someone wants to find out if the system worked as designed. So an engineer calls up the relay (or goes on site) and extracts the fault record. This record has the date and time of the event, as well as how long the fault current was present (and something called oscillograms, which is basically the sinusoidal waveform showing exactly what the voltages and currents looked like). This tells us exactly how long it took for the relay to see the fault, and how long it took the breaker to trip. Sometimes breakers are slow, and this logging record would reveal that. Sometimes relays are set improperly, and the record could help determine that as well.

These records are stored electronically in the relay, and can be extracted and stored in a computer. But they are only used for "after the fact" investigations, and are not stored in a mainframe anywhere.

I hope this helps.

-- Anonymous, June 04, 1999

Let me provide a more immediate example of logging, date, and time stamping. Actually, it's a repetition of something I've talked about before.

(And honest, folks, I'm not picking on the nukes - it's just a good example.)

At a nuclear facility, during a plant transient, critical process monitoring and system responses are logged in a "sequence of events", or SOE, computer. This logging takes place in increments of 100th's or 1000'ths of a second, depending on the granularity of the particular SOE system. I don't remember design spec's on the one I was familiar with.

Regardless, if you've ever been in a nuclear plant control room when hellzapoppin', things are happening very quickly and the only way to capture all of the relevant data in the sequence a particular action occured (either automatic system response or operator induced action) is by using an SOE computer. And depending on the nature of the transient, sometimes the only way to get a grip on what's happening, and more importantly, why, is by reviewing the SOE computer printout (or screen).

The mere sequence in which specific automatic systems responses occured may dictate proper recovery response, and in any case, aids in poste mortem analysis of the event and root cause determination.

So, yes, data logging and time / date stamping is particularly important in this case.

The SOE printout provided the "smoking gun" in many events that I reviewed during my previous life.

-- Anonymous, June 04, 1999

rick,

in the president' commission report on the tmi incident the sequence of events are listed in seconds from the time of the incident e.g., 36 seconds after 4:00am the feedwater systems pumps tripped.

when the feedwater pumps tripped the plant systems shut down the steam turbine and the electric generator it powered...the incident was 2 seconds old.

reactor 2 scrammed...the incident was 8 seconds old.

my question is this? is this data retrieved from the soe computer?

do i understand you correctly when you say that this data may be used to 'get a handle' on the situation? is this computer listed as 'mission critical' by the nuclear power plants?

why do factfinder and engineer say 'only' time and date stamping problems when they refer to an operation that seems, to the layman, at least, critical to the resolution of an incident?

or am i misinterpreting what you are saying?

-- Anonymous, June 05, 1999

marianne, no, you are not misinterpreting what I'm saying.

SOE recorders / computers serve as the "black box" (to use an airline analogy) of nuclear power facilities. TMI 2, to the best of my knowledge, had no dedicated SOE recorder. Recommendation D3 of the Kemeny Commission report states:

Montioring instruments and recording equipment should be provided to record continuously all critical plant measurements and conditions.

-- Anonymous, June 05, 1999

By the way, marianne, I haven't forgotten about your earlier request. I will get that copy of the report out to you by the end of this coming week. I need it for something I'm doing on Tuesday.

-- Anonymous, June 05, 1999

Dan:

Thanks for your response. It has been helpful to me.

Does everyone agree then that the date information that is stored in the relay is "nice to have" and not "necessary for power?" It sounds like a date problem in the system you describe would not trouble the power system. It would only trouble those who wish to investigate what happened. It sounds to me like the EMS could deliver anything for a date and it wouldn't be bad for operation of the relay. Is this true?

Rick:

The system that you have detailed presents a case in which there is a critical need for logged time and date information. This would be the type of system that would need a careful system test, wouldn't you agree? Do SOE's run on mainframes? Are SOE's custom built by individual entities? Do you have any idea what kind of coding languages are used to program SOE's? Are SOE's present/important to any other kind of power plant (non-nuclear)?

-- Anonymous, June 07, 1999

Usually in a sequence of events you are looking at the times not the date. What you are looking at is the order of events and the time between them. Time to the millisecond is important. The date really isn't. You tend to know that anyway and can easily either write it down on paper or create a text file and put the date in it when you store the data.

It wasn't too long ago that all you had were dropped targets and you had no idea of what happened when anyway, unless you had a light beam OSG. And those didn't have any date and time on them.

Sometimes we do store data on mainframes (at least at our company) but I can't work up a lather over it. Several years ago they threw out a bunch of data I had from the early 70's because no one used the tapes in several years. So they just recorded over it. Keeping data for a long time is not (depending on the company) a major concern.

Reporter a question: What is your concern with mainframes?

-- Anonymous, June 07, 1999

As I recall from my SCADA days long ago, there is a big difference between LOGs and SOEs. Logs are recorded by the control system or SCADA, or EMS, to about 1 or 0.1 sec accuracy. SOE recorders, on the other hand, are not control systems, but just record signal events at about 10 ms or better accuracy; these SOE records are used for post disturbance reviews (precise analysis after a problem or outage occurs).

-- Anonymous, June 08, 1999

Mainframes are typically complex systems. If there are no compex systems in a critical path, then that greatly reduces risk of finding a Y2K problem in that path.

-- Anonymous, June 08, 1999

Reporter, Don't have a lot of time here, but I believe that it is only fair to provide you with a quick response on the log issue. It is my opinion that there are many cases where logging of dates is a very important function - in nuclear, EPA data logging, etc. So I want to make sure that I do not dismiss this issue when I state that most Y2k problems are minor. Please note that my experience has been that we are upgrading non-y2k compliant data logging problems. Also note that it is my experience that most of these data logging date bugs could be dealt with administratively if needed (again, my opinion), so they are not truly "severe" in my mind. But in some cases, they are certainly important. Regards, FactFinder

Regards

-- Anonymous, June 10, 1999