Fears of Y2K Power Failures

Energy: Utilities have found making repairs to their equipment less complicated than expected.

By ASHLEY DUNN, Times Staff Writer

Robert Haverkamp's first take on the year 2000 problem at the San Onofre nuclear power plant was that it would be a daunting task, akin to finding technological needles buried in a huge haystack of steel, silicon and wire.

There were 190,000 separate devices at the sprawling facility south of San Clemente--any one of which might fail because of the computer glitch. But as he and his team began to dig in, they quickly realized that it was not quite the haystack they had imagined. Out of 190,000 devices, just 32,000 were electronic. Of those, only 2,900 used computer chips, made up of just 356 types of items, such as circuit boards.

After four months of work by 45 people, they discovered that only about one-third of the 356 types were date-sensitive, and just 38 types, amounting to a few hundred devices, needed replacing.

"It's easy for me to feel comfortable now," Haverkamp said. "But I spent many long hours wondering if we had found everything. You always worry you're going to find something big. We didn't."

Power--or the lack of it--has been one of the greatest fears about the year 2000 problem because of the potential for an outage to bring a modern society to its knees. But as more power utilities have moved deeper into their repairs of the computer glitch, they have discovered that the problems have been easier to fix than expected. Although there are still concerns about the completeness of the repairs, the possibility of a large-scale power failure in the U.S. has begun to fade. The utilities have benefited from the repetitiveness of devices in energy generation, the rare usage of a date in their equipment, and the predominance of older electromechanical devices--antique pieces of technology that are immune to the year 2000 problem. Even at the most complicated power facilities such as San Onofre, the search for defective pieces, though laborious, has turned out to be straightforward. The actual repair of devices has been relatively simple, often amounting to replacing some computer boards and updating software. A March survey of more than 3,000 power generation and distribution companies by the North American Electric Reliability Council, a trade group that coordinates electric utilities across the country, reported that the industry is about 75% finished with its repairs--roughly on schedule to meet its June 30 deadline. The biggest electric utilities in California--Pacific Gas & Electric, San Diego Gas & Electric, Southern California Edison and the Los Angeles Department of Water & Power--are well ahead of the curve and are confident that the most populous and power-hungry state in the union will have few problems Jan. 1. For all the encouraging news, some concerns still persist, particularly about small, rural operations. Sen. Robert F. Bennett (R-Utah), chairman of the Senate Special Committee on the Year 2000 Technology Problem, wrote in Science magazine last month that the 1,000 rural electric utilities in the country may lack the resources to fix their problems in time. "A prolonged nationwide blackout is not likely," Bennett wrote. "Local and regional service outages, however, are a distinct possibility." The industry has generally lagged behind the fast starters in millennium bug repair efforts, such as banks and financial service companies, despite its critical position in the transition. The General Accounting Office reported last month that there are still "significant risks" in the electricity industry because of its slow start, though it was basing its report on a survey conducted in November. The most recent data paint a more encouraging portrait, showing about 90% of utilities finishing within a few days of the industry's June 30 deadline. Just four expect to complete their work as late as October. One critical facility that will finish late is PG&E's Diablo Canyon nuclear facility southwest of San Luis Obispo. PG&E said the job will be delayed because it is waiting for a planned shutdown of the plant in October to complete its work. Overlying these concerns is the nagging question of whether the utilities have found all the chips and programming code affected by the millennium bug that are scattered across thousands of square miles or buried deep in the innards of complex facilities. "The biggest effort in this was just making the lists of everything," said Steve Malowney, year 2000 project manager for Sempra Energy, the parent of Southern California Gas and San Diego Gas & Electric. "Everyone was so worried; we looked really, really hard." The root of the problem for the electric utilities has been the digital monitoring and control equipment that is scattered through their systems. There are not a lot of different devices, but they are placed in ways that can make them difficult to find. Unlike the banking and financial services industries, which have largely had to deal with fixing software, the utilities have also had to grapple with this complex hardware problem. Their computer controls and so-called embedded systems suffer from essentially the same problem as software. The year 2000 problem stems from the long-standing use of two digits to represent years in computer programming. The system works fine as long as all dates are within the same century. But in 2000, computer systems could become confused over the ambiguity of the date "00," which could be read as either "1900" or "2000," leading to the possibility of malfunctions or miscalculations. The glitch includes several other date-related problems, such as the miscalculation of the 2000 leap year and the use of the date 9/9/99 as a file marker in programs. Kevin Seay, year 2000 program manager for PG&E, said the utilities were fortunate in that much of their equipment is old. For example, the company's fossil-fuel-burning plant in Humboldt Bay was built in 1963, long before the digital revolution. A large portion of the company's power comes from hydroelectric plants and those, he said, are "actually a little easier because they are the lowest-tech of the bunch." "For better or for worse, a lot of our equipment isn't that high-tech," he said. Seay said the company, now about 95% finished with its mission-critical work, budgeted $107 million for its year 2000 repairs but believes the final price tag will be less because there was less work than expected. At the privately operated Applied Energy Services generating plant in Redondo Beach, no repairs have been finished, but the work is relatively limited because of the age of the plant, which dates to the 1950s and '60s. Steve Winters, year 2000 coordinator for the plant, said the entire project involves only about 150 components. "Everything has been ordered," he said. "We're just waiting to install the pieces." Over the last 15 or so years, utilities have been adding digital components because of their versatility, reliability and record-keeping ability. But the numbers are still small, largely because there has been no great need to add such sophisticated and expensive controls. Eric Trapp, year 2000 program manager for Southern California Edison, said that out of the 440,000 components in Edison's power system--made up of 2,300 types of devices--only about 7% to 10% are digital. Of those, a minuscule number--about three dozen unique types--were found to be defective. John Ballance, manager of Edison's power grid dispatch and operations, said that even in those digital devices, time is almost exclusively used for record keeping and that even if some of the components had failed, the power would not have gone off. "The control systems really don't care what day it is or what year it is," he said. "They are just there for logging purposes." Edison, which has completed about 85% of its year 2000 repair work, estimates it will spend about $8 million less than the $80 million it budgeted. One cautionary note about the numbers game in the power system is that even though there are few devices, the failure rate of those digital components with some time function can be relatively high. Haverkamp at San Onofre found that about 40% of the time-based digital devices he found were defective, though none could have forced a shutdown of the plant. Although repairs at most power systems are well on their way to being completed, the biggest fear is that a small problem in a distant part of the power network could cascade through the system. The power grid is an interconnected web made up of generating plants, substations and high-voltage transmission lines. Supply and demand must be constantly matched to avoid brownouts and outages. California's grid is fed by utilities as far away as Utah and Oregon. In 1994, damage to a high-voltage line in Southern California caused lights to flicker in places as distant as Alberta, Canada. The connections among power systems can also help when there is high demand, by allowing utilities in various areas to share power. The balance of electric power in this state is overseen by the California Independent System Operator, a not-for-profit corporation created by the state in 1996 after the deregulation of energy. Cal-ISO is in charge of monitoring the state's network of high-voltage transmission lines. From control centers in Folsom and Alhambra, it measures the energy load on the lines every four seconds through sensors placed around the state. Cal-ISO began operations in March 1998, and its computer systems were built to handle the year 2000 transition. Above Cal-ISO is the Western Systems Coordinating Council, which handles the interconnections among states and regions. California has agreed to follow a WSCC recommendation to raise its power reserves from 7% to 20% through New Year's weekend to avoid any shortages. "That's a huge amount of reserve," said Dennis Fishback, Cal-ISO's chief information technology officer. To avoid the cascade effect of problems in one area migrating to another, Cal-ISO has also agreed to another WSCC recommendation to reduce power transmissions among regions to the minimum, thus providing a buffer in case of a sudden overload. Fishback said the steps taken by Cal-ISO and the WSCC will help to ensure that even if there are problems, they will remain local and temporary. "We really don't expect any significant problems, if any problems," he said. He added that one of the most powerful allies in ensuring the steady flow of power Jan. 1 will be Mother Nature herself. Winter is a season of low electricity use, and midnight is one of the most stable periods for power generation. In addition, California's location 20 hours from the international date line will give the state plenty of warning of possible problems with power equipment. "It really couldn't have happened at a better time," Fishback said.

-- Norm (nwo@hotmail.com), May 09, 1999

Answers

Excelent news for CA. 1 down, 49 more to go.

CR

-- chuck, a Night Driver (rienzoo@en.com), May 09, 1999.

Does the dim in the title refer to the lights in your brain Norm?

Enquiring minds want to know.

-- Andy (2000EOD@prodigy.net), May 09, 1999.

Andy, I am curious why Norm is subjected to a hostile attack? Reviewing the majority of his posts and have not yielded any negative statements directed at you. Lets see a rebuttal to his post that leaves out the derogatory comments.

-- pdirac (pdirac@hotmail.com), May 10, 1999.

pdirac,

butt out - Norm has history buddy...

-- Andy (2000EOD@prodigy.net), May 10, 1999.