NERC's "Generic Inventory"

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

I'm relatively new to this forum and realize that the following issue might have been raised and answered before, but I'll hazard it anyway:

I recently downloaded NERC's "Generic Inventory of Components that may be Y2K Susceptible," available at http://www.nerc.com/y2k/ This inventory runs to 44 pages of very fine print and lists over 3600 items, ranging from complete systems to individual components, that NERC considers might be vulnerable to Y2K. Each system or component is named, its function briefly described, its place in the overall scheme of things noted (admin., business development, generation of electricity, or delivery [T&D] of electricity), its priority indicated (1 being high, 2 being middle, 3 being low), its Y2K compliancy status recorded (yes, no, or blank [unknown]), and its vendor(s) listed. Some items (systems or components) are identified as compliant, some as noncompliant--but the vast majority are simply listed as "blank," that is, their compliancy or noncompliancy is unknown. This includes many systems or embedded components involved in the generation and delivery of electricity: turbine and generator monitors and controls, controls of distribution capacitor banks, voltage regulators or controls, recloser controls, etc. For some of these items NERC apparently doesn't even know who the vendors are!

Now, I've read carefully both of NERC's quarterly reports (Sept. 17, 1998 and Jan. 11, 1999); in both, and especially in the latter, NERC asserts that power companies are finding fewer Y2K problems than expected and that very few problems impacting the generation and delivery of electricity have been found. I find these report statements incompatible with NERC's own inventory list. As a matter of logic, both cannot be true. If companies are indeed finding in many of their embedded digital components that compliancy is not even an issue (i.e., no internal calendar function in any of the chips), then why aren't these items simply deleted from NERC's inventory list?

I have tried to find ways out for NERC. For instance, perhaps NERC is simply not updating its inventory list. But then why bother to post it online if it is so outdated and misleading? Another possibility I considered, based on the excellent report analysis by Rick Cowles and Bonnie Camp, is this: since roughly one-third of U.S. power companies have completed 30% or less of remediation/testing of their mission-critical systems, perhaps NERC is simply waiting until all the data is in before "filling in" the blanks on all those components whose compliancy is not known or stated on its inventory list. That is, perhaps all those companies that have actually checked their voltage regulators, recloser controls, etc., so far have found few or no Y2K problems, but NERC wishes to wait until the laggard companies have reported their test results before "closing the book" on these embedded components. Still, even that theory makes one wonder why NERC is optimistic in its quarterly reports that there will be no serious Y2K disruptions of power, given that so many companies have yet to check so much equipment. After all, in one paragraph buried late in its last report, NERC warns power companies NOT to rely upon type (sample) testing of embedded digital components in distribution systems. (It seems to me that companies obviously also shouldn't rely on type testing of embedded components in generating facilities and transmission systems!) If NERC itself recognizes the dangers of generalizing with regard to embedded systems, then how can it reliably present any "conclusions," optimistic or otherwise, in its quarterly reports, given (again) that so many companies have yet to check so many embedded systems individually?

Any enlightenment on this issue will be appreciated. And let's hope that NERC's optimism is justified and that we all will have "enlightenment" next January!

-- Anonymous, February 01, 1999

Answers

I am not certain about the source of the inventory, I couldn't get it to open on the computer I am borrowing. I SUSPECT that it is derived from the Testing,Results,etc tab of the NERC reporting form.

Go to the NERC site and checkout the reporting form. Note that on the testing tab, the reporting of results is voluntary. I suspect that some utilities reported inventory early on, and then realized that it was redundant with EPRI reporting, which is more useful to the utilities. Don't fret about this list, it probably isn't up to date and has been made obsolete by a redundant one at EPRI.

-- Anonymous, February 01, 1999


If EPRI maintains a Y2K inventory list available to the general public, does anybody have a link to it? I visited the EPRI website and almost everything seems to be closed except to members (at $75,000 a throw, as I recall).

A couple of further points about the NERC "Generic Inventory." If NERC is indeed not bothering to update this list for the public, after putting it online, such sloppiness and irresponsibility bother me. Also, it's my understanding that NERC first put this list online in October, AFTER it issued its first quarterly report on Sept. 17th saying that power companies were finding fewer Y2K problems than expected. The inventory published online in October directly contradicts NERC's statements in its Sept. report. Finally, even the last NERC report, issued Jan. 11th, must have been largely prepared at the end of December or beginning of January, a mere two months or so after the October online publication of the Generic Inventory. Unless marvelous strides in knowledge were made in those two months, I can't see how some of NERC's latest report statements can possibly be justified. In short, the evidence seems compelling that NERC has shown less honesty and integrity than one could wish.

-- Anonymous, February 01, 1999


Yes, the EPRI site is closed to members only. This allows members to discuss problems found without betraying propriatery info or giving vendors advertising fodder. The problems found are discussed in detail at the EPRI conference (doesn't take very long - NERC is portraying things accurately. I am almost bored testing with nothing to report.) NERC doesn't need the details of each device, just a summary of the totals and that data is available to you (I believe you can see each utility's response with the names removed). The summary is taken from these responses. How can NERC be irresponsible for not giving details that it doesn't have. As far as how NERC gets the summary info that no sig problems are surfacing - I don't have a clue. I do know that a NERC rep attended the symposium that I participated in where my company got together with all municipals, coops and cogens for info exchange. I also assume that the regions of NERC all stay in touch with member Y2K project teams. I know that if NERC asked me that I would tell them that I have found no sig probs.

This NERC report is a hassle. It takes away from testing time. I've already gotta report to about three different bosses, each wanting a status report, PLUS sharing results with others via EPRI. If the gov't makes something optional, and it is inferior to an existing forum I would rather eliminate the extra work. Perhaps it is a valid point that NERC should remove the inventory from the report. BUT, do you think there is a chance that the public would cry "conspiracy" if the unused inventory showed up missing?

Let me close by saying that I did search the inventory and found a plethora of items that have no compliance status in the NERC inventory, but HAVE been tested by multiple utilities and are Compliant OR Ready.

-- Anonymous, February 01, 1999


CL,

What have you been testing that is making you so bored? Sounds like good news to me.

NERC can be irresponsible by portraying the situation as being much better than they can accurately determine. They have reasons to do that. That's my take on the broo-ha-ha.

What's a "sig"?

Regarding the search you did on the inventory of items that have no compliance status; This is the sort of data that sounds important to me. Why wasn't it included in the report? What's up with that? If the items are compliant or ready why not report it?

Steve

-- Anonymous, February 01, 1999


If you have 100,000 embedded chips to test and after the first 1,000 you get board because there is no sigprob found, what conclusions can you draw from that knowing that chip number 98,999 kills the system? Keep plugging away because we all are depending on you not to be board.

It reminds me of the movie Andromeda Strain. Where the scientist is looking endlessly at computer permutations to kill the virus and the 1 in a million cure is found but passes right by her gaze because she is in some kind of catatonic state.

Keep drinking that black coffee buddy!

-- Anonymous, February 02, 1999



Well, "cl," I do appreciate your responses, and, as somebody who doesn't own a portable generator, I certainly hope that you are correct. I'm willing to accept that you are a hardworking power company employee who believes that his company and various other power companies belonging to EPRI's forum have found relatively few Y2K problems thus far. Unfortunately, many of the 7800 power companies in the U.S. don't belong to EPRI's forum. Also, you must realize that those of us trying to analyze information objectively in the public domain can do only so much with undocumented assertions by anonymous sources. We need documented status reports on specific systems and embedded components. I realize that you may be constrained by your company's policies, and I certainly don't blame you for that. The "public" policy of many power companies on Y2K is an unfortunate combination of arrogance and secrecy--not the wisest course, in my opinion.

But that brings us back to NERC, which has been charged by the DOE with the task of not only monitoring the power industry's progress on Y2K but also providing important, reliable information on this topic to the general public. To have released a report in September that expressed optimism and general knowledge, and then to have posted online (in the public domain) in October a "generic inventory" that clearly revealed that NERC simply didn't know the Y2K status of much electrical equipment, continues to strike me as highly irresponsible. If the October inventory list was accurate--and I can't imagine that NERC would suddenly decide to post a list that was inaccurate, especially one that was so unsettling--then the Sept. quarterly report pretended to knowledge that NERC simply did not have. The responsible and honest course would have been to admit in that Sept. report that "we don't know but we're doing our damnedest to try to find out."

To the present: if NERC has gathered more information in the past couple of months on the compliancy status of thousands of items in its inventory list, then NERC should take the time and trouble to update that list. It doesn't matter what EPRI is or isn't doing; NERC is an entirely separate organization, with its own task clearly outlined by the DOE, and with a moral responsibility to keep updated and reliable information posted in the public domain. More simply, if the generic inventory was important enough to be posted online for the public in the first place, then it is important enough to be kept updated. After all, how much trouble is it to type, under the column heading "Compliant," either "yes" or "no"? If compliancy has been found not even to be an issue with that particular item, then simply delete that line from the list. Not a terribly time-consuming or difficult task.

My uneasy speculation is that NERC still doesn't really know the compliancy status of many items on that list, despite a January quarterly report that (even more so than the Sept. report) claimed to know all sorts of things. Indeed, one can find misleading statements and internal contradictions in the January quarterly report itself. The excellent analyses by Bonnie Camp and Rick Cowles have already revealed how misleading "averages" can be in such a report. Worse, the "conclusions" presented upfront, designed for "soundbite" press releases and given to the National Press Club in Washington, often don't jive with information buried later in the report itself. The press release material doesn't contain NERC's statement (buried 60 pages or so in the report) that it can't absolutely rule out a "catastrophe" (NERC's word, not mine), nor does the press release contain NERC's warning against reliance upon type testing and vendor statements (this warning was also buried late in the report, in a single paragraph on testing methodologies for distribution systems). Or consider NERC's upfront "conclusion" that few, if any, Y2K problems found thus far would interfere with the delivery of electricity to customers; in the body of the report, NERC admits some Y2K problems with EMS/SCADA systems, which obviously could interfere with the reliable delivery of electricity to customers. We all know about the 1996 Y2K EMS test at Hawaiian Electric (HE) that failed dramatically. (HE has since fixed this system, of course). A few months ago I had an email exchange with Phil Hystad, the principal designer/architect of three EMS and SCADA systems used by major power companies. Hystad claimed that no EMS/SCADA systems (his or anybody else's) had any Y2K problems! When I directed him to the HE story, documented in the June 2, 1997, issue of "Newsweek," he at first tried to deny that any such outfit as Hawaiian Electric (Honolulu) even existed! When I proved to him that HE did exist, and even referred him to their systems analyst, Wendell Ito, Mr. Hystad rationalized that HE's failure must have been the result of an "outdated" Rockwell system. Thereafter I heard nothing from Mr. Hystad. I have a friend who lives in Seattle and I note that Seattle Light, for one, has significant Y2K problems with its SCADA system; fortunately, remediation and testing are slated to be completed by this June. Well, no doubt Seattle Light is also using an outdated Rockwell system.

The point, again, is that NERC and others have a public responsibility to present reliable, documented, detailed, up-to-date information in the public domain on crucial issues involving Y2K and the power industry. Jefferson noted that a democracy can't survive without an informed citizenry; that observation has broad ramifications.

-- Anonymous, February 02, 1999


Don, I agree with you about the disturbing generic inventory list on the NERC site. I list has not been updated since Sept. 22 of 1998. Look at the doc info section in adobe reader. The original title was "Sample Electric Utility Inventory1.XLS". Maybe its not even a real document! What disturbs me most is that this database looks like most of the public vendor compliance databases I have seen, half filled out, most of the compliance statements missing. If these are indeed the latest and greatest information, we are in trouble. I know it is hard work to even get them filled in completely. How could you assess your overall compliance status if the picture was this fuzzy?

Mike

-- Anonymous, February 02, 1999


Moderation questions? read the FAQ