What about manual back-up systems?greenspun.com : LUSENET : Electric Utilities and Y2K : One Thread |
Repeatedly, when I've talked with people in charge of running our local power companies and their y2k projects, I've heard, "Well. If worse comes to worst, every part of the system has a manual back-up." Will those systems work, and if not, why are these people (who really sound like they know their business) saying they will? It seems obvious we simply aren't going to make it if the embedded systems problem is as big and knarly as everyone's saying it is. Isn't it time to start paying closer attention to manual systems (contingency plans) and making sure they're as ready as they can be - just in case?
-- Anonymous, January 03, 1998
Harris,I have been receiving that exact same response from power station personnel that I have spoken to. For older stations, they are correct. Unfortunately, what they forget is that most personnel that can remember how to run the station manually (before it was automated) have retired or have been given the golden handshake through down sizing. After all, who needs people when your plant is automated? For modern power stations, manual operation should have been built into the basic design. To manually run a power station with numerous generating units, remembering that most (if not all) of your controls and safeties are off line, would require a large amount of people. You would need enough for 24 hours a day, 7 days a week until all automatics were restored. The response to that was Well, all the office staff can give a hand. Now lets consider sending office staff (who have most likely never been in a power station let along worked in one) over to help run one power station. How long would it take to train all the people required to perform their various tasks? You would have to think about workers compensation insurance. Demarcation issues with unions. Would the office staff want to work in a power station anyway? Well, they might not have a choice in that matter. Who would be chosen to go? Human Resources, Information Technology, Secretarial, Document Control, Management (ha), Stores, Distribution, Transmission, etc. Where would all the people come from that would be needed to run ALL your power stations? I am currently contracted to a power corporation who handles at least 7 major power stations and numerous minor ones. Each of these has several generating units.
Sorry for the length of this response, but it hit a nerve.
Regards, Kim
-- Anonymous, January 10, 1998
Kim,I appreciate your well thought response, even if Harris did hit a 'nerve'. It's difficult for anyone who has not actually worked in a large power generation facility to appreciate the complexity of these facilities and the total interdependency of *all* mechanical, electrical, and instrumentation systems required to make the facility run.
You are absolutely correct about the resource issues related to manual operation, particularly in newer facilities. As you point out, even in the older facilities, the 'history' (the people that could keep the place running with bubble gum and duct tape)is gone.
And you touched on one area that I've yet to see anyone seriously discuss at length (although I've touched on it a few times myself) - union issues. I can tell you for a fact that union issues are going to be a player in the equation as contingency planning for Y2k begins to take center stage.
-- Anonymous, January 11, 1998
Hello All,As I have said previously, a power station would or should have manual override operation in-built into its design, albeit old or new. The designers have to consider normal failure and breakdown of equipment, after all every thing wears out sooner or latter. On the other hand, if the designer put all their faith in technology and discounted any form of manual human intervention, your in trouble! Maintenance crews would or should have plans to repair systems that inadvertently breakdown. It is a part of normal engineering life to have the odd breakdown or two. Maintenance crews will take breakdowns in their stride and fix them as and when needed and overall do a good job. Coming from an electrical controls background, I know that most devices or systems that fail can be fixed one way or the other, given money, time and resources. You may have the money (may not) but you are running out of time and can you find the resources you may need? They might be over at another states (or countrys) facility/plant fixing them up! - - Now, your power station was design to use one or two types of PLC, they were used for almost everything. This makes perfect sense, as it minimises operating systems to be learnt and used, the number of different hand held key pads and spares to be held in stock. One of the PLC types is affected by a Year 2000 problem and you need to correct or fix the 20 or 30 or 40 or 50 or more you have in your power station. Do you have enough maintenance staff to look at all the PLC quickly? Do you have a budget that will cater for extra staff/contractors and overtime? You will not be the only one who uses this PLC. Can your vendor supply substitute devices? Are they compatible with the existing input (and output) devices? Do you have to re-wire part or all of your control cabinets? Will they fit in were non-compliant PLC were? - - Another power station automatically takes emission samples from smoke stacks, cooling water, ash treatment etc. and passes the results to the local Environmental Protection Authority for authorisation. If you can no longer report these readings to the EPA, how long do you think the EPA would let you keep running your power station for, not long I bet! OK, you could get people to take samples, analyse them, create a report and pass that on. Would that conform to your quality management procedures? Would the EPA accept manual readings? - - A fully automated, triple redundancy system controls a power that, in its design, does not require any human input to its operation. Only an occasional maintenance would be required by staff or contractors. Normally, if one system fails or gives problems, the second would take over and then the third if the second followed the first. Let us say that all three systems have an identical or similar designed and the first controlling system encounters a year 2000 problem and failed or performed below specification. The second system senses the problem with the first system and takes over and immediately encounters the same Year 2000 problem as the first system and fails. The third and last system senses that the second system has failed and takes control and immediately encounters the same Year 2000 problem. Possibly in a very short time, the triple redundancy system has just been made redundant through its own design. You may find that such a system may not have ANY manual backup incorporated in its design. Scary, is not it. - - In short, Yes most power stations could be operated manually, but I would not like to try, it could get very messy. One guy I work with has likened the situation of manually controlling an automatic power station to that of driving a car blind folded (you, not the car). You know that you will hit something, the only question is, when and what (or who)?
Brian, my brain hurts quote from Monty Pythons Life of Brian
Regards, Kim Smith
-- Anonymous, January 12, 1998
FYI -- Submitted for publication; feedback desired; it's on this thread's topic. -- Thanks!
The False Assumption Lay People Have That All Utilities Can Operate In a Manual-Bypass Mode in January, 2000 - Water Supply Implications
Roleigh Martin
1/25/1998
(Text is 1047 words)
Some people have read my article, "The Year 2000 Embedded Systems Threat to Core Infrastructures" published either in the January/February 1998 issue of the widely circulated, Year/2000 Journal or on the Minnesota Software Association website at http://www.msa.org. Others may have visited my web site at http://ourworld.compuserve.com/homepages/roleigh_martin (where links to the prior two articles exist). Among those who have followed my concerns and to whom I have had the opportunity of speaking with, some tell me not to worry because the core infrastructures will be able to operate in a manual, bypass manner if their automation controls fail because of Year 2000 embedded systems problems.
"Embedded systems" has been best defined by the U.K. Institution of Electrical Engineers as "... devices used to control, monitor or assist the operation of equipment, machinery or plant. 'Embedded' reflects the fact that they are an integral part of the system. In many cases their embeddedness may be such that their presence is far from obvious to the casual observer and even the more technically skilled might need to examine the operation of a piece of equipment for some time before being able to conclude that an embedded control system was involved in its functioning." *
The Year/2000 embedded systems problem deals with embedded systems that are subject to failure because their software is not Year/2000 compliant.
Core infrastructures refers to the utilities that provide us with electricity, clean water, natural gas, waste removal, phone service, and so forth. Reliable, large-scale distribution of electricity is the most important issue, because almost all other utility services depend on it.
The embedded system Year 2000 problem with core infrastructures concerns equipment with embedded control devices used by utilities that will operate improperly (causing equipment failure or worse) after the turn of the century, or in some cases after 1/1/1999 or even other dates.
I have received in-depth reports from three investigators who work inside the electric utility industry and whose job is embedded systems Year 2000 compliance. All three of these individuals report that my web site material is either accurate or when not, that my problem is that I am missing additional factors to be concerned about -- that is, that I am understating the problem! The material I have received is being reviewed by lawyers whom are helping me on a voluntary basis as to how to safely make use of the material. However, one quote from the investigators needs to be released now. One investigator reported that a new HVAC system failed in year 2000 testing and wrote:
"New HVAC machines (as well as others) do not always contain manual override for 'on' position. Work-arounds are impossible without costly replacement. Lead time on replacement of such machines has put several utilities into guaranteed non-compliance."
A HVAC is a heating/ventilation/air-conditioning system. Without HVACs working, rooms containing electronic sensors and heat-sensitive data processing equipment can overheat and start malfunctioning.
Now, as always, what is the impact of non-compliance? In some cases, it only impacts the rollover across centuries, but in other cases it can impact ongoing operation in the new century (even if it was shut down for the rollover hour) for various reasons too technical for purposes of this article.
[continued in next post]
-- Anonymous, January 25, 1998
FYI -- Submitted for publication; feedback desired; it's on this thread's topic. -- Thanks!The False Assumption Lay People Have That All Utilities Can Operate In a Manual-Bypass Mode in January, 2000 - Water Supply Implications
Roleigh Martin
1/25/1998
[Continuation -- Part 2]
Bottom line: it is foolish not to be concerned about this problem if you and your family value electricity. Some people act as if it would not be analogous to preparing for a winter campout in a unheated cabin. But these people forget the water issue--most winter camping sites have hand-operated water pumps. If your family is dependent upon city water, then chances are that electricity is needed to maintain water pressure for the water has to be pumped up into water towers so that gravity thereafter can maintain water pressure.
In conventional power outages, the problem is typically caused by events external to the power plant and a city-wide outage is very rare. In conventional power outages, backup equipment identical to the failed equipment is available for replacement. In a Year-2000 outage, an outage could be caused by equipment inside the power plant where the same Year-2000 noncompliance flaw exists in the replacement equipment. If this is the case, compliant-replaceable equipment might either have to be custom made or purchased and it might not fit in exactly right with the other equipment interacting with the failed equipment. In other words, a variety of new equipment might be needed with long lead times required. A Year-2000 outage could mean a prolonged period without electricity for the area unless the power grid can supply the area previously served by the downed power plant--but if an abundant number of other power plants across the nation have the same problem? You get the picture.
Although most water utilities probably have backup generators, for how long a period are they prepared for? How long can your city go without city-wide provided electricity and maintain water pressure? In a sub-zero weather in January 2000, how long would a period of a lack of water pressure take before becoming a city-wide disaster? Even if the electric utilities had their act together and water pressure was okay, will the water utilities' equipment work after 1/2000 so that the water is clean?
Steve Gravelle, in the 9/10/1997 issue of the Cedar Rapids Gazette newspaper in Iowa, wrote about the loss of electricity and its' impact on water. "With no heat, residents in older homes and mobile homes turn on their water to prevent their pipes from freezing. But with no electricity, pumps can't replenish the water towers, and Marion soon runs out of water. And when power is restored, the entire water system must be flushed and treated to prevent contamination." Quoting Ned Wright, Linn County Director of Emergency Management: "'The power may be out for five days,' Wright said, 'but you can't drink the water for seven.'"
It is the above reasons that principally drives me to continue this campaign. I live in the coldest state in the nation (a Minnesota city has the record as the coldest city in the nation!). As long as the experts refuse to discuss their preparations and the inspection, remedy, and fallback measures being taken--if any, I am seriously concerned. You should too if you value electricity and city-provided water.
----------------------------------------------
* Special thanks to David Spinks for providing this definition of embedded systems. Thanks also to Karl E. Vogel for his help with some of the other definition paragraphs.
About the author:
Roleigh Martin, M.A., is a software engineer consultant who has written for The Year/2000 Journal and the Minnesota Software Association's publications on the Year 2000 embedded systems threat to core infrastructures. He has a web page at http://ourworld.compuserve.com/homepages/roleigh_martin where these two articles can be read. Prior to his data processing career over 20 years ago, he was a research sociologist employed at various research centers and was a consultant for a National Science Foundation grant on system modeling. Over the years, he has done various freelance articles.
-- Anonymous, January 25, 1998
One of the reasons that we, modern man (no sexist meaning intended), automated and are automating EVERTHING is that generaly machines can do the job safer, quicker, better and usually with more reliability.When you are bringing a power station on line to contribute power to a grid, a number of parameters need to be considered. Please correct me if I'm wrong.
Firstly, the generation units need to powered up. For a large fossel fuel power station you will need a donkey 'gen set' big enough to supply power to start up the larger units. Gas Turbine units can usually be started under their own power (gas or diesel). If there is no donkey 'gen set' on site and the grid is down you'll need to get a 'gen set' in a hurry. Once started, you need to match the phase output of your station to that of the grid, easy enough done, just use the instruments on the local control panel. You can SEE what phase is what. Next you need to match the load/output of your power station to that of the grid. If you don't match the output power correctly you can trip out the other power stations on the grid (effectively shutting other power stations down). To match your power to the grid, you need to know how much power (watts) to output. This power matching is usually controlled from a central control station. A SCADA system is used to FULLY control the remote and local power stations, ie the generating units. OK, if you can't SEE the amount of power to output, the control station can tell you over the telephone. If the phones are working? So do you just twiddle a dial and hope that you don't trip out other power station or do you just switch off???
Now, put yourself in an aeroplane, black out ALL the windows, turn off your all instruments that SEE the outside world. Start your craft, taxi down the runway and take off. Sounds simple, hope that you don't bump into anything and I do hope that you can land safely (if at all)?
Yes, you can control a power station maually, but like the plane, you know you can fly it but you also know that very soon you are going to crash into something.
Any other comments?
Regards, Kim Smith kimsmith@arach.net.au
-- Anonymous, February 26, 1998