Contact: Eugene F. Gorzelnik
efg@nerc.com
April 9, 1999

Electric Systems Exercise Y2k Telecommunications Capabilities

Today the electric power industry of North America conducted a Year 2000 drill simulating the partial loss of voice and data communications needed to operate the electric power grids of the United States and Canada. This drill was facilitated by the North American Electric Reliability Council (NERC), who is coordinating Y2k preparations of the electric power industry and monitored by the U. S. Department of Energy. At no time during the drill, were electric customers affected by the exercise.

Overall, we can report that the drill was a successful exercise of back up voice systems and manual procedures needed to operate in the unlikely event of a loss of communications due to Y2k. Twenty-two regional security coordinators participated in the drill, in addition to about 200 electric utilities across North America. Initial reports indicate more than 2,000 electric utility personnel participated in the drill.

Typical drill scenarios included loss of external voice systems and real-time data systems used to monitor and control electric power systems. Additional aspects of the drill included partial loss of data acquisition and control computers in the energy control centers. Several hundred generating units participated in the drill, including coal, gas, oil, coal, hydro, and nuclear facilities. Over 500 critical electric facilities around the continent were staffed with field personnel using backup radio, privately owned telephone switches, microwave, and satellite voice systems. These personnel relayed critical information, such as power flows, voltage and frequency, back control centers to allow critical system operating tasks, including generation dispatch and energy interchange. Some of this information from substations and power plants was updated every 15 minutes to allow the simulation of continuous monitoring and control.

Additionally, the drill included interarea coordination across regional boundaries with NERC and DOE personnel in the DOE Headquarters Emergency Operations Center in Washington, DC. This drill was extremely valuable for the training of personnel and practicing backup communications procedures. Many lessons learned have been identified in the initial assessment conducted late this afternoon.

• There were some incorrect phone numbers identified.
• Geographic coverage of low and high frequency radios will require relocation of some antennas.
• Personnel require additional training on the use of satellite voice systems; however, some of these systems have been only recently installed, leaving little time for training prior to the drill.
• Some congestion of voice traffic on some of the backup systems indicates a need to establish and practice voice traffic priorities.
• Some of the backup voice transmissions were noisy.
• A small number of the backup systems did not work as planned and will require further checkout prior to the next drill.
• There were severe lightning storms in the Midwest today associated with a storm front; this weather did temporarily interfere with some of the radio systems.

The electric industry backup communications drill was successful in that it provided an opportunity to exercise backup communications systems and procedures. As the industry prepares for a second major Y2k drill on September 8-9, they will implement these lessons learned.

-- Anonymous, April 09, 1999

Answers

Rick, the success of this RADIO communication drill could be short lived; have you seen the predictions for the 11-year solar cycle? This summer begins the 1-2 year period of coronal mass ejections (CMEs) which I'm sure, being in the power industry, you are aware of. They are expected to be "heavy" this year depending on where we are in the orbit when CMEs occur. Do you know if this radio based contingency will be adversely affected? I guess we'll all know after the Sept. test if the sun does its' thing...

-- Anonymous, April 09, 1999

Rick,

My sister lives in Montana and the electric blinked off once and then later was off for over a half hour. When she called to report it the individual stated she had no idea of my sisters location after she gave her the mailing address and physical location of her home. While on the internet an individual in California I was communicating with suffered a blackout that lasted about a half an hour later in the day.He stated he hadn't had a blackout in several years.

Were these blackouts related to the testing? Was the inability of the individual at my sisters electric provider to locate her location for assistance when her electric went out related to the testing going on?

Thanks for your article, but I'm confused as to exactly what went on and the report that I am reading. I guess it could just be weird occurences at an inopp. time.

Bob Wood

-- Anonymous, April 10, 1999

In Southwest Montana, also had a brief (10 second) power outage Friday morning, April 9. Coincidental?

-- Anonymous, April 10, 1999

Mr. Wood,

Living here in Florida, I would be hard pressed to say with any degree of certainty that your sister in Montana suffered an outage or blackout resulting from tests run by the electric industry. Given the fact that each of the people you mentioned were only off for about half an hour, more or less, they most likely experienced the brief interruption as a result of an electrical fault in the distribution system caused by trees, vines, animals, auto accident, mylar balloons, storms, wind, lightning or any number of uncontrollable events that occurs every day in this country. The blink she saw before the lights went off indicates the distribution breaker in her local substation attempted to clear the fault before automatically reclosing to restore power. Clearly, the fault remained so the distribution breaker completed its cycle and locked open. This is when the field crewmen have to locate and clear the fault manually. If power was restored in half an hour, I congratulate her local utility they almost as good as ours!

-- Anonymous, April 10, 1999

"...the success of this RADIO communication drill could be short lived..." - Bob Hampton

Bob,

I doubt that the radio systems will be affected significantly, if at all. The report mentioned that hand-held radios were (to be) used during the drill.

Most hand-held transceivers (aka HT, for handy-talky) operate in the VHF/UHF range of frequencies. This is line-of-sight communications, because the frequencies they use are too high to bounce off the ionosphere, as long range, or HF, transceiver emissions do. The radio emissions just punch through the atmosphere, and are lost to space.

With this in mind, it is safe to say that, with the aid of automatic (not necessarily computer or embedded-chip controlled) relays, aka repeaters, HT users can talk over long distances.

Ham radio operators do this daily, and have at their disposal a vast network of VHF/UHF repeaters to use in the pursuit of their hobby. Hams in some areas have cooperated with each other to link their repeaters, so a ham in one state can talk to another ham in another part of that state, or a neighboring state. There is one network in the SW US that can allow a ham in Arizona talk to hams in Texas, with no more than a little battery-powered HT.

When you think about it, first responders are rarely affected by solar disruptions, because their radio communications are also in these frequency ranges. Because their services are vital to life and property, their systems are backed up to beat the band (pun intended). If they can operate during severe CME-induced com disruptions, so can the electric industry.

Hope this helps clear up a few things.

LP

-- Anonymous, April 11, 1999

I appreciate the individual efforts of our electric utility workers to prepare for interruptions associated with Y2K, however I find this report of a successful Y2K drill to be almost laughable. Having done some investigative research on the readiness of the electric industry, I discovered several internal documents on the NERC's ftp site that would cause a few eyebrows to raise, including the Y2K Drill Preparation Guide for the April 9th drill. The guide clearly states that the intention of the drill is to instill public confidence through success, i.e. The drill was designed to be successful from the start in order to calm public fears. According to this guide, careful measures were taken to test the systems to be used during the drill, up to two months prior to the drill to ensure there were no surprises and that successful parameters be established with regard to the NERC's final report. The NERC states that the drill was not to be complex and that they want to have a successful and meaningful story for publication. Individual utilities were allowed to determine the extent of their own testing, but pointed out that they should make maximum use of simulators and a separate staff assigned to work the drill rather than normal operations staff. The NERC also stated in the guide that the drill should be conducted so that it would not interfere with normal operations. Some Drill... The NERC's report of a successful drill went off without a hitch. It's amazing what you can find on anonymous ftp sites. For instance, The NERC has modified how monthly Y2K status reports are handled. With the comments given by the Secretary of Energy Richardson during a January 11th press conference emphasizing the need that "all mission-critical electric operating systems be Y2K Ready by June 30th, 1999". (which is only 77 days away), The NERC decided to change how they report the industry's readiness. They now exclude the numbers released to the DOE and the public in cases where utilities have a non-conforming Y2K program. The numbers from their confidential database can also be found on their ftp site and if you do the math, you can see that what is being reported to them is in sharp contrast to what is being reported to the public and the government.

-- Anonymous, April 14, 1999

Brandon,

It would be presumptuous of me to attempt to wordsmith the NERC Y2k Drill Preparation Guide here and clearly, the document you refer to uses some language that, in the public view, may be considered unfortunate. Particularly in view of the difficulty with which we, in the industry, are attempting to calm fears of utter chaos after the rollover. This is not an attempt on my part to re-interpret the document for you or this forum. This forum is replete with reasonable people and opinions and with frank and open debate.

However, having said that, I would take issue with your characterization and uses of the terms investigative research and internal documents. It is my understanding that this document has been available on NERCs Y2k web pages for quite some time and would require minimal effort or investigative techniques to acquire and download. Likewise, I do not think the guide was intended to be considered internal as your post is not the first reference to this document I have read. Anonymous ftp sites? Well, if you say so.. but if I am mistaken about the availability of the guide, please let me know.

The industrys intent for the drill was very correctly characterized by Ms Bonnie Camp in her comments on April 5th located here: http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000gin Personally, I do not believe it was the intent of the NERC document to conspire with the North American utilities to mislead the public. But hey, thats just my singular opinion and you most certainly have the right to reach your own conclusions.

For myself, I would have preferred a guide that was designed to be successful and not disrupt normal operations as opposed to a guide designed to fail. Further, to fail to prepare for a drill is, as far as Im concerned, the same as failing to prepare for an emergency.... absolutely inexcusable. At my company, we simulated the loss of both voice and data communications used for routine and EMS/SCADA operations on the corporate as well as statewide level. For us, things went quite well. Across the country, several difficulties were encountered and problems discovered. These shortfalls can be seen within this forum. Therefore, I would not characterize the drill as going off without a hitch.

Heres wishing you and everyone a hitchless New Years!

-- Anonymous, April 15, 1999

JT, Thanks for the response to my communique'. I sincerely hope that the electric industry will be ready for any rollover problems that may come their way. God forbid that we take a situation like NYC in '77 and magnify it 1000 times. In response to some of your obvious questioning of my terminology, I'll try to clarify, even though it is really irrelevent. First, ftp sites (file transfer protocol) can be created for either a user logon or anonymous user access. The nerc's ftp site has both. It was through this medium that I have viewed several documents that in my opionion would "appear" to be internal, for example: photocopied credit card statements for the president of the NERC, Michehl Gent showing all his transactions. (which were laden with computer purchases) and things of that nature. I just take it for granted that when a document says "for internal use only" that that could be considered an "internal document". As far as my investigation into the electric industry is concerned, you have misinterpretted one example of public access information as the sole source of that term. I assure you that literally hundreds and hundreds of hours have been spent reading, calling, e-mailing, etc. involving every level of technical and administrative personnel. That is, in my opinion "investigative research". Allow me to clear up another point while I'm at it, I stated the the "report" went off without a hitch, not the drill. The intention of the drill was to instill public confidence through success. So how could we expect to hear anything less than a happy ending when that was the idea all along? The drill served two purposes . One of which was to report how great everything is concerning the electric industy's Y2K remediation efforts. I mean, come on. Have you looked at the survey figures in the NERC's database? You may want to believe the NERC wouldn't conspire to mislead the public and that's fine. I just have a hard time putting faith in an organization that clearly states in black and white that they are not going to report the data from utilities that have non-confroming Y2K programs to the DOE and the public. I applaud the results of the drill at your company and appreciate the input. It is just my opinion that when conducting a drill, the intention should not be so that you have a "successful and meaningful story for publication", but to thoroughly check out your systems for problems that need attention and train personnel for actual emergency scenerios. I agree with you that failing to prepare for a drill is inexcusable. Perhaps, you would like to respond to my claim concerning the NERC's Y2K Readiness Assessment Exception Reporting Policy. I encourage a simple explanation as to why the agency that was commissioned by the Department of Energy to oversee the electric utility industy's Y2K remediation efforts and report the status thereof would adopt a policy that is clearly political in nature and undermines the very task that it was asked to do. I look forward to your response.

p.s. If you are ever in Dallas, Let's have some lunch and some laughs.

-- Anonymous, April 16, 1999

Brandon,

Thank you for the clarification and explanations provided in your response. Please let me affirm that it was not my intent to call your serious efforts to gather information into question. Nor was it my purpose to challenge your expertise in gathering data and with it, hold the industrys feet to the fire. Most importantly, I would never question your right as an American electric consumer to ask those questions and demand answers. It is the most important reason I frequent this site. I would like to think I am simply on the other side of the fence in my opinions. I cannot say the same for Yourdons or Norths playground.

It was, primarily, my intent to convey some measure of caution in the interpretation of the guide document and offer an alternate view. It is unfortunate that a presumption of deceit could even be interpreted from this document in the first place. I would speculate that Mr. Gorzelnick feels the same. For me, often when the debate becomes frustrating in some forums, I am reminded of a line in a poem I read years ago, the author forgotten:  and thus, in his considered view, what did not suit could not be true.

Who, what, and how Y2k readiness statistics are calculated and published? I am at a loss to offer any simple explanation where the exceptions policy is concerned. The variables themselves are exceedingly complex and deal with numerous issues such as de- regulation, environmental issues, capacity availability, system economics, timing, resource availability and service reliability to name a few. I regret that I cannot help you but perhaps you may prefer to get your answer from the source. I am confident that an e- mail or letter to Mr. Gene Gorzelnick would get you a response. You do have a right to know.

Clearly, at issue here is the potential for widespread disruptions within the electric industry resulting from Year 2000 problems and the cascading effects throughout other industries as a consequence. No one has a greater interest in seeing to it that power will flow before, during and after the rollover than the electric utilities themselves. Personally, I am witness to the seriousness with which my company and most companies within my region (FRCC) have dealt with this issue. Honestly, I find the level of cooperation and coordination among utilities in our region astounding. But then again, it is important to us. In the view of our corporate task force, it is critical to the relationship that we enjoy with our customers that their questions do not remain unanswered. We frequently participate in or sponsor well advertised Y2k workshops for residential as well as commercial customers. While our spokesman was selected for his experience and comfort with public speaking, he was also selected for his ability to respond and convey answers to complex issues simply, openly and honestly. My perception and belief at this point is that our customers consistently demonstrate confidence in our ability to meet this challenge. We expect to meet this challenge by June 30, 1999.

Again, I do choose to believe that a nationwide plot to deceive and mislead the public does not exist. Perhaps its too much for me to accept that hundreds or thousands of American utility managers and employees would acquiesce to such a directive. As a practical matter, I simply cannot see DOE providing a better level of oversight than NERC given the resources and contacts required. But, as my 12 year old daughter likes to point out. Daddy you dont really know everything do you!

Hey lunch and laughs sound terrific Dallas sounds better!

-- Anonymous, April 17, 1999