Analyses for Complex Systems of Systems (DOD)

greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread

I am a poster on the Timebomb forum and we had a poster saying that he was a DOD "what if" contingency planner and that there was going to be problems. It was never resolved whether he - she was who they said they were but I posted the link below for their opinion. No one commented and a gentleman Emailed me to post it here. Now I have little education on the power grid and would like to have some informed opinions myself.

So here we go.

Y2K Analyses for Complex Systems of Systems:
http://cr-iiacfs1.army.mil/army-y2k/y2kelectric90224/tsld001.htm

Y2K Analyses for Complex Systems of Systems: Electric Power Systems in North America February, 1999 DRAFT

CNA

PPT Slide

"In the electric industry, the extensive computer and control systems that operate power plants, the relays and circuit breakers that protect the system during short circuits, the communications systems that allow operators to control system elements at remote sites, and the energy management system computers that control the flow of electricity across the grid are all susceptible. Software and electronic hardware glitches could cause any of those systems to malfunction resulting in the unexpected opening of transmission lines, outages of generation, or loss of system control elements. "

NERC Reliability Assessment 1998-2007, September 1998

Page 3

Briefing Agenda

Characteristics of the North American electric power industry that relate to Y2K.

January 11, 1999 Y2K Report of the North American Electric Reliability Council (NERC).

Lessons learned from the July 1996 Electric Power Outages in the Western United States.

What should be done in the time remaining

Page 4

Characteristics of the Electric Power Industry Relating to Y2K

Deregulation and the impact of market forces on the industry have made it more difficult to respond to Y2K:

Vertical disintegration and new market entrants make information sharing and coordination harder and less universal. Extensive regional and national interconnection has resulted in widespread long-distance sale of electrical energy ("wheeling"), straining transmission lines and reducing generation capacity margins.

Page 5

Characteristics, cont.

The electric power industry has a long tradition of voluntary self-regulation.

In 1968 electric utilities formed NERC to help its members assess reliability of generation and transmission systems, to prevent blackouts. The DOE has tasked NERC to coordinate Y2K transition of the electric power industry.

NERC is transforming itself into an organization with compliance enforcement powers - but not in time to deal with Y2K.

Page 6

NERC January 11, 1999 Y2K Report to DOE

Tone of the report, the second of planned quarterly reports on industry Y2K preparation, is one growing optimism:

"With more than 44% of mission-critical components tested through November 30, 1998, findings continue to indicate that transition through critical Year 2000 rollover dates is expected to have minimal impact on electric system operations in North America.."

Page 7

NERC Presented 3 Issues:

Not all facilities or systems will meet the target date of June 30, 1999.

But the industry will still be able to provide reliable, sustained service in the Year 2000.

It has proven very difficult to perform integrated testing of electrical voice and data communications supplied by external providers.

Distribution systems use few electronics but are radial, with limited redundancy options.

Page 8

NERC Has Lowered The Y2K Compliance Standard

Last year NERC reduced the standard :

"Y2K Ready means a system or component has been determined to be 'suitable for continued use into the Year 2000.' Note that this is not necessarily the same as Y2K Compliant, which implies fully correct date manipulations. Consistent with practices across other industries, the NERC assessment process has adopted the term Y2K Ready and does not use the term Y2K Compliant."

NERC, Preparing the Electric Power Systems of North America for Transition to the Year 2000., September 1998.

Page 9

Average % Complete of 3-Step NERC Members' Y2K Program

Page 10

CNA Assessment of the NERC January 1999 Y2K Status Report

End-to-end testing of complex grid interconnections is very difficult to do without creating outages over very large areas.

Not acceptable to the power consumers. Could severely damage electrical equipment.

Integrated testing of externally provided voice and data communications with the electrical industry is very difficult.

Leaving potential vulnerability of a critical interface unknown.

Page 11

CNA Assessment of the NERC January 1999 Y2K Status Report, cont.

Inability to test the vulnerabilities of the interdependence between the electrical industry and communications suppliers makes it hard to know what will happen if there are failures.

It may be possible to gain insight about possible electric system behavior under Y2k contingencies by examining a recent large-scale outage. . .

Page 12

July 1996 Power Outages in the Western United States

-- A case Study --

Page 13

North American Electrical Interconnections (Grid Graphic)

Page 14

The Events: July 2-3, 1996

At 2:24 PM the 345KV transmission line between southwestern Wyoming and southeastern Idaho shut down automatically due to a short circuit.

A second later a parallel line tripped in error, disconnecting two large generating units from the grid.

After 23 seconds, a voltage collapse began in Idaho and eastern Oregon.

Page 15

July 2-3, 1996, cont.

After 35 seconds the Western grid serving 14 states, 2 Canadian provinces and northern Baja California separated into 5 electrically separate islands.

To avert major damage to generation and transmission facilities.

Service to 2 million customers was interrupted for from several minutes to 6 hours.

On July 3 a replay of these events began at 2:03 PM but was manually contained.

Page 16

DOE Analysis, August 1996

Initial cause was a planned-for event, a "double contingency" that should have been contained in the local area:

A short circuit between a sagging line and a tree caused the first transmission line to trip. The parallel line tripped due to a faulty relay which interpreted the first event as its own.

2 of 4 generating stations were disconnected from the local grid, introducing frequency, voltage and current instabilities into the area.

Page 17

DOE Analysis, cont.

The instabilities were not contained, and a cascading effect began in the Western Interconnection grid.

The grid could not cope with the instabilities because it was moving an unprecedented volume of cheap Canadian hydroelectric power generated from prior abundant rainfall into the Northwest, Rocky Mountains and California.

A result of market competition. Stressing transmission lines.

Page 18

DOE Conclusions, cont.

Need for better procedures and information systems:

To restrict Available Transfer Capability ("wheeling" between grids) when necessary. To produce faster local load shedding in order to avoid islanding when transmission systems become overloaded.

Worry that WVA/VA 765KV grid interconnection planned for 1998 won't be available until 2002.

Making mid-Atlantic states vulnerable.

Page 19

CNA Y2K Analysis

Industry deregulation since 1996 may have made the electrical power grid more vulnerable to Y2K:

Competition produces far more inter-grid power wheeling, stressing transmission stability beyond industry modeling and planning July 1996 power outages in the West are an indicator. Unbundling of generation, transmission, distribution and brokering makes coordinated exchange of information and action more difficult.

Page 20

CNA Analysis, cont.

The industry voluntary self-regulation regime, with extensive deregulation and divided Federal and state government roles, may not be responsive enough and fast enough to deal with the Y2K issue.

Gartner Group now ranks the power industry and its vendors Level 3, meaning a projected 80% chance of at least one Y2K-induced mission critical systems failure in 50% of the nation's electric utilities.

Page 21

CNA Analysis, cont.

The industry systems for modeling and analyzing contingencies emphasize continued operation in spite of the "most severe single contingency",

In July 1996 the Western Interconnection grid crashed with 2 small contingencies.

These systems are far less capable of dealing with multiple and dispersed contingencies.

But Y2K failures are almost surely going to be multiple and geographically dispersed - even if not catastrophic individually.

Page 22

CNA Analysis, cont.

The industry strategy of isolating failures may fail because few participants may be healthy enough (Y2K compliant in every important respect) to execute it.

Moreover, the strategy assumes that all required fixes or workarounds to the initial failure - e.g. an offending tree or relay - can be made quickly, thus allowing the system to reconstitute itself in hours or days.

What if the Y2K fixes take weeks?

Page 23

What Should be Done in the Time Remaining?

Electric power is the most important industrial sector from a Y2K perspective:

Little else works if there is no power. Restoring systems may take days or weeks.

It is unlikely that we will know what will happen to electric power until we experience the Y2K transition.

Public Education and Contingency Planning are key steps for the time remaining.

Page 24

Public Education

It is vital that a national program to educate the public be mounted early in 1999:

To foster and encourage not only individual preparation but also neighborhood and community preparation. To prevent anxiety or panic: there are already publications and web sites forecasting millennium catastrophe. There are indications that uninformed people may hoard essentials and take money out of the banking system and the stock market.

Page 25

Public Education: The Message

The message:

We won't know for sure what will happen until it happens. Don't panic: government and industry are well along in contingency planning to prevent serious problems. The key is effective community preparation. As we do for hurricanes, flooding, earthquakes and other natural disasters. We will help you make preparations for yourself, your families, your neighbors and your community.

-- Anonymous, June 16, 1999

Answers

The document is genuine.

-- Anonymous, June 16, 1999

"We will help you make preparations for yourself, your families, your neighbors and your community... "

Shouldn't leaders of countries and states be supplying this kind of direction now?

-- Anonymous, June 16, 1999


It seems to have been created by an organization called the Center for Naval Analysis (www.cna.org)

A major league military think-tank, also instrumental in organizing the Nval War College scenario brainstorming sessions.

-- Anonymous, June 16, 1999


Brian,

Isn't this interesting stuff. They are really looking at all the angles and finding a lot of weak links. Will the chain hold? Who knows. But if it does, it will hold due to not being stressed beyond these many weak links. Talk about crossed fingers! The happy face good news folks don't like these reports, but personally I find more and more of this sort of thing popping up, and it never seems to get more optimistic. Do you suppose this is why the military and DOE are putting together some pretty serious contingency plans right now. And I don't mean just buying a little portable gen set here and there.

-- Anonymous, June 17, 1999


For me this briefing was the Rosetta Stone for understanding what higher levels of the federal government believe will happen to the power grid, and its overall impact on the nation.

CNA is a very prestigious U.S. Navy Think Tank. It usually deals only with matters of fundamental strategic significance -- what are perceived as serious threats to U.S. strategic security. CNA has (or had in the '80s) a joined-at-the-hip relationship with the Hudson Institute. This organization is best known for its founder, Herman Kahn, who wrote extensively on fighting nuclear war. I am familiar with CNA because I participated in contract work with CNA as an employee of another firm.

This briefing indicates that CNA is briefing senior DOD officials that the power grid will probably fail due to Y2K problems. Remember that CNA is a MILITARY think tank. This tells me that they view the potential problems as great enough to consider developing a domestic military contingency. The need for such a contingency would be based upon a conclusion that grid disruption will be widespread enough and long enough to place extraordinary stress on conventional emergency services.

Why isn't this information being made known more widely? Probably because neither CNA nor anyone else knows what level of readiness the electric utilities will reach by Jan 1, 2000. Putting out a national call for preparation prior to knowing how serious the problems will actually be is likely being seen as a move that could disrupt the ability of the utilities to get as much work done as possible in a "normal" environment. This briefing hints at a higher level fear of people getting wacky regarding the more serious implications of electric grid problems.

No effort to prepare the public will begin until authorities have a comprehensive view of (1) What is Y2K ready/compliant as of June 30th and (2) What can be expected to be ready by Jan 1st. According to a local official here where I live, the public awareness campaigns will begin in July. My guess is the information will be disseminated through local organizations with the message coordinated at the national level. The federal government will try not to make nationwide statements in order to give the semblance that only localized problems are expected. Any nationwide warnings will be handled by the Red Cross. Red Cross' teaming with Ralston-Purina also indicates we may will a public/private program to spread the message. The message and coordination will be through the federal government, but its influence will be masked so as not to raise the level of the Y2K problem to a national crisis. (Biggest fear of the high level decision makers is 24 hour coverage on CNN and MSNBC)

The message will be upbeat, with the same touch and feel you would expect if we were drumming up support for our Olympic team. (Or maybe selling War Bonds)

As for the optimistic words we hear from the industry connected experts on this site -- they probably are providing accurate information. However, remember the CNA briefing begins with the assumption that you can bring down the grid with two distinct and minor events. I think everyone here would agree that all utilities connected to the grid will not score 100% on this test. And from CNA's perspective, even a 98% means a blackout.

Remember now, this is nothing more than speculation. But based upon what I am reading and hearing, it seems the most logical at this point in time.

-- Anonymous, June 17, 1999



With regard to the original post, it can be found at:

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000w5S

(Be warned however. There were well over 150 responses to this. You may not want to go there)

This, however, may have been the most interesting response:

"This is one of the worst spoofs I have ever seen. Too many obvious mistakes and gaffes to waste much time on.

I am Director, Year 2000 International Security Dimension Project, US Naval War College. No pretend names from me. Bio at both NWC site and at personal site at geocities.

Check out our site at http://www.nwc.navy.mil/dsd/y2ksited/y2ksite.htm at War College, or, for .mil-domain challenged, our duplicate at geocities: http://www.geocities.com/ResearchTriangle/Thinktank/6926/y2ksite.htm

We're actually doing what this spoof artist claims to do.

The future is transparency.

-- Thomas Barnett (barnettt@nwc.navy.mil), June 11, 1999.

(What does "The future is transparency" mean? I still don't know.)

-- Anonymous, June 17, 1999


Bill,

I agree with you. While the general public statement is "Nobody Knows", there are nevertheless some pretty competent folks getting together and outlining some probability scenarios that are pretty grim. And they're not doing this based on the "official" industry reports coming from such outfits as NERC.

-- Anonymous, June 17, 1999


Folks

Thanks for your information, FM it should be pointed out the thread you are reffering to is the DOD "what if" guy and not the information posted above. This has been kicking around for sometime and was discussed on the Timebomb forum before but no one could provide more than speculation on the report. Unfortunately the links are long gone.

-- Anonymous, June 17, 1999


You're right, Brian. I kept a bookmark of a discussion of it in April:

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000jn1

I remember it being discussed there at least a month before that. It is quite authentic.

-- Anonymous, June 17, 1999


The Naval War College site can be accessed here: Year 2000 International Security Dimension Project Summary. It concludes with
Bottom line of Naval War College effort:

Understanding that there is a tremendous gap between the public face many corporations and governments put forward on this issue ("we will have it well in hand") and the private fears and concerns expressed by many information technology experts (ranging from "global recession" to "apocalypse 2000!"), we want to explore this topic in as systematic a fashion as possible. We don't pretend that we'll end up with all the answers, but merely a sensible read on what's possible, how governments and companies are likely to respond across a range of scenarios, and what the USG and DoD should be prepared to undertake in response to Y2K's global unfolding. In short, while we're not interested in unduly hyping the Y2K situation, we are interested in exploring the "dark side" potentials because, frankly, that's what we get paid to do as a research organization that serves the U.S. military.

The roster of participants in the varius colloquia mentioned is quite impressive. This is not a trivial effort.

-- Anonymous, June 23, 1999



Moderation questions? read the FAQ