Our plan for LA: Set clocks ahead in Oct to fool computers

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

www.cajunelectric.com/y2k.htm

Cajun has developed contingency plans to address potential problems caused by Y2K. In October 1999, Cajun also plans to roll forward 1,730 MW of fossil generation to the year 2000 to mitigate the possible affects of the Y2K transition dates. These plants will remain in an advanced timeframe until the Spring of 2000 when they will be rolled back to the actual date.

Cajun Contact: Will Day, Y2K Program Coordinator

(225) 291-3060 (voice)

(225) 296-1746 (fax)

Problem SOLVED!!! Now, that wasn't so hard now, was it?

-- Will Day (info@cajunelectric.com), August 13, 1999

Answers

Many electric companies are doing this. OPPD in Omaha, NE has already rolled some of their generating facilities forward.

www.oppd.com

Of course they did type testing. Not too confident about them.

-- Justalurker (nope@nope.nope), August 13, 1999.


Will, is Cajun Electric (or any LA supplier) going this deep on the Sept. 9 drill?

The following are examples of  scenarios, time horizons, and 
resources committed for the September 8-9 drill:

 Loss of remote terminal units (RTUs)  Blackstart radio check  Pipeline stations to operate on manual for 2-3 hours  12 cooperatives to talk together and with VP by satellite phones  Managers and supervisors work with employees at substations; focus on communications  Start at 1400 hours on 9/8 for dress rehearsal until 0200 hours on 9/9  Loss of normal communications; test satellite links; process computer in nuclear plant  Hospitals  Customer drill will simulate a total outage of electric supply; will lose air conditioning; will actually go on backup power noon to 1600 hours  Test contingency plans and command center  Satellite communications and backup radios; distribution lockout/tagout will be manual  Man 20 substations for dress rehearsal; lose some generation; put some generation that is on standby on-line  Full rehearsal during day; expand communications drill to include customers; use exit strategy; communications drill with PJM at midnight  Use of ham operators and 800 MHz emergency management radio system  Fail business systems; use cellular phones; coordinate with Emergency Management Agencies (EMAs), some customers; problems with generation and transmission; 800 people, 9 pm 9/8 until 2 am 9/9  Post rollover checklists/ verifications  Simulate loss of RTUs, energy management system (EMS); put standby combustion turbine online; 10 pm until 2 am  Coordinating with city EMA; exercising media information center; include one substation failure (non-Y2k) at 11:30 p.m.; potential load curtailment exercise; drill 50 of the 250 people available for December  Contingency response plan deployment; monitor power systems in Europe to see how they respond; some local simulated failures; 10-20% of workforce out that night  Daytime exercise with call center; communications drill in PM; set up emergency centers; loss of load scenario; plant trip; loss of environmental data; loss of bottom ash system  Tabletop exercise on loss of load (less than anticipated load); communications drill; operating, Information technology, management, and facilities management personnel; deploy people for midnight rollover; open Y2k information center; media and government communications; drill call centers on outage calls and inquiries  Involving 150 public utility districts (PUDs, small customers); radio communications with them; broadcast capability to get information out to them; involving hydro units  Communications checks with neighboring systems; reading tie flows; calculate manual area control error (ACE); media at pool control center; media liaisons will do their real job; simulate loss of lines and generation; simulate curtailment of transactions; 8 am to 4 am on 9th; practice regional and NERC call groups  Loss of communications and use of human RTUs  150 people in substations; full mobilization at 10 pm 9/8; report data until 2 am 9/9; 250 people  Exercising business processes; information centers; customer care center; corporate communications; 7:30 pm start  Loss of RTUs; phone systems; staffing blackstart units and key substations; roll out media relations; 400 people; use training simulator set to New Year conditions  Simulate flood; staff emergency center; test communications with cooperative and municipal customers  Loss of communications from control center and local dispatch centers; use plane for an airborne repeater station to bridge a communications gap; how to convert human RTU data to useful information  6 pm 9/8 to 2 am 9/9; staff all blackstart units; frequency points assigned to plants; verbal talk through of blackstart procedure  Power marketers participating from their emergency center; coordinating with generation facilities and exercise contingency plans  Tabletop blackstart drill; staff substations in evening and simulate loss of communications; coordinate with emergency management agencies  Staff contingency event centers; staff substations; operate with third level backup EMS; set up data collection process; include nuclear facilities; high voltage contingency at 3 am  Business-side scenarios will not be known ahead; military and communications companies also are exercising (what kind of interactions with them?)  Use satellite communications for control area operations; simulate communications failure with neighboring system, fossil plant, and nuclear plant; activate information center  Test communications with host control area; not full dress rehearsal (municipal transmission dependent utility)  Loss of communications and islanding of system simulated; over midnight deploy human RTUs  Contingencies in fuels area; loss of fuel and communications; loss of public switched network; be able to communicate with key accounts; condensing contingency procedures and rollover into several hours  Full dress rehearsal; second tier business systems will be included  Try report by exception  Training a second shift to be ready  Loss of supervisory control and data acquisition (SCADA) system, customer call center, communications; weather scenario; inviting media and customers to participate  Including EMAs, simulated events; full deployment of staff; use of backup communications, human RTUs; control area to control area communications using satellite phones; reserve activation, load shedding, and other contingencies simulated  Business systems; simulating ice storm and materials management; facilities management  Emergency operations center; ham radio links; satellite phones to regional center; emergency broadcast tests; maintain a media center separated from the control center; executive management not at control center  Primary and alternate teams; trained and drilled in June; August 12 rehearsal, including a shift change; unannounced drill scenarios; simulate loss of generation, load, reduced SCADA and EMS and communications; will do shift change during September drill; customers, media involvement  Need to nail down good list of telephone numbers  Information system (IS) and information technology (IT) will run four different drills; run an IS and IT war room; will operate during rollover and drill; several hundred people ready to respond from IS/IT area  Board members invited to participate  Loss of electronic data exchanges; process to check accuracy of human reported information; some surprise scenarios built in  Some large customers also setting up drill activities and want to interface with their power supplier  Including cyber and other threat scenarios in the drill  One customer providing backup phone system to power company to allow it to stay in communication with the company should there be a loss of public switched network  Prepare for customer calls into customer service center  How to measure effectiveness of drill? What processes are effective?

2. What are the biggest challenges? How are you solving them?

 Large staffing issue  just the size of the activity; plus keeping regular jobs going  How to measure effectiveness of drill processes?  Are utilities planning to contact vendors and other critical suppliers during the drill  Phone lists and radio frequency call groups  Training of personnel; training in communications  Avoid over-committing resources; need additional people available  Lots of people involved in drill that would not normally participate  Prepare a timeline with control messages; criteria of what the observer is looking for; rated on the spot  Hospital - cost of lost revenue; large dollars involved; testing communications and backup vendors  Getting unions to go along with the resource staffing plan  Training personnel in event management and reporting processes  Exchange of status information in company is harder; much harder now to include external companies and agencies  Coordination of activities around midnight; confusion around midnight if simulated failures  Over 600 people; logistical management challenge; labor union cooperation (volunteers for duty on New Years Eve)  Assign project coordinator and team; assign accountable persons in each area  Verifying manually obtained RTU data by plugging into EMS manually and comparing it to real time data  Not able to get all the same people during drill and New Years rollover; large percentage will be same and have same assignment; training pack for each substation; give to personnel one week ahead as assignment is known  Involved a wide range of departments and personnel in some tabletop exercises  Take snapshots of system data and verify with data reported from field; keep good logs with time stamps; follow-up with debriefings soon after drill  Simulate movement of generation based on human RTU data; then see how generation actually moves on real basis  How to take care of major (all) customers; notify them of what is going on; every service area has at least one radio station that people listen to for emergencies; will provide timely statements to those radio stations for public and customers to hear  Have satellite providers give us a report on satellite phone usage during the drill  Beacon getting a lot of attention  Most are planning to fuel trucks prior to key period; should this be practiced; resolve logistical issues  Overcome some challenges by keeping the drill simple  Could we do a simultaneous load test on the satellite phones?  Will code of conduct be suspended?

3. How are you involving distribution systems, IPPs, power marketers, EMAs, and others?

 Getting distributors involved this time; setting up organizations and radio communications with them; additional new training  Working with EMAs; utility personnel in those facilities with radios  Simulating loss of natural gas; coordinate response with loss of communications  Concern with run on gasoline at end of year  need to fill up early  How do you stay up all night? Let people doze/nap, as appropriate, make sleeping arrangements available  Exception reporting vs. roll call to verify communications still OK.  Every communication needs to say This is a drill before and after every statement  Monitor CNN and local TV station  Functional verification checks  Prioritize information requirements  Logistics of moving people around



-- interesting, commendable (scenarios@drills.tests), August 13, 1999.

(Dan 7:25 KJV) "...and think to change times and laws:"

-- Mark Hillyard (foster@inreach.com), August 14, 1999.

Moderation questions? read the FAQ