This is the FINAL chapter written about the electricity industry and Y2k, from the perspective of insiders. Special thanks go out to all those who contributed: Malcolm Taylor, FactFinder, The Engineer, CL, and Jim Cooke. Enjoy the final installment: CONFESSIONS

========================================

Dan's comments:

Although Y2k was never really a major threat to the industry, there were a few things that I thought were screwups....

1) One of our major switchyards (with hundreds of devices on site) was assessed for Y2k. It boiled down to only ONE piece of date-sensitive mission critical equipment. Since it was a one-of-a-kind security system, we had to pay a contractor several hundred dollars to come out a y2k test it for us. The device passed the test fine (Oct. 1998), and, as we always do, we placed a sticker on it to identify it as Y2k ready, with a warning to not alter it without contacting the Y2k team.

Fast forward a year later...while making final preparations, I mention to the technician who'll be working on site the night of the rollover, to keep an eye on that device. "But that thing should be fine, because they just replaced it," he says. I was miffed; this tech works on site every weekday, was there for the original test, and I told him to let me know of ANY changes on site. "Why didn't you tell me it was up for replacement?" I charged. "I dunno", he replied. His punishment was that he had to physically stand in front of that device and babysit it during the rollover. :P

2) While presenting Y2k status to a group of senior citizens, one of our neighboring electric utilities was explaining all their preparations..."Everyone will be on call December 31," the spokesperson said. I squirmed a bit in my chair, knowing that this was a bit inaccurate. Then came this from her: "And in addition, every one of our employees will be carrying a pager that night." WHAT?? 6,000 PAGERS? No way, man.

3) One axiom of the pessimist crowd was that "smaller" power companies were falling behind the big companies in preparation. I didn't believe this until early 1999, when presenting to a group of the smaller utilities. To my chagrin, it was true. Many of them couldn't even get started because some management person thought that Y2k was some kind of hoax. Lucky for them they don't use digital technology much, which allowed them to catch up rather easily.

4) We decided to do a replacement program of our substation satellite clocks. I felt good about kicking in money to help pay for and coordinate the work. In February, 2000, long after the rollover, I get a call from a technician at one of our substations. "There's this clock that doesn't seem to be working." Sure enough, we had missed one. The kicker was when I called the manufacturer to complain that they hadn't informed us. They faxed me a letter sent to us, dated October, 1999, that described exactly how to fix it. It was sent to a specific person in our purchasing department. I called that person several times, and never got an answer back. It's very embarassing when a supplier has better records of equipment than your own company does.

5) One local city decided to do a Y2k drill on their "911" backup system. So they shut of the main power breakers, waiting for the backup generators to kick on and power up the building. Nothing happened. Upon further inspection, they found out why: The cables from the generators had never been connected to their electrical system! Evidently it had been this way for over a year...

6) The legal folks were in general a great help to us, and rarely intervened in our Y2k remediation efforts. One area that kind of rubbed me the wrong way was when we were encouraged not to talk about a few things: Dont make any predictions about what you think will happen; When asked, What should I do to prepare?, direct them to the Red Cross web site; and dont discuss your personal preparations.

The reason the last admonition bothered me was that I was not at all hesitant to tell people that I did ABSOLUTELY NOTHING to prepare for Y2kno extra cash, water, nothing. Would not that have spoken volumes about my confidence in the electrical system?

==========================================

The Engineers Comments:

We didn't have too many screwups, at least that I know of. One, in the general sense, was I think we should have started earlier then we did. I brought up the subject a number of years before it became an "official" concern but no one else thought it was a big deal. Later one of the VP's went to a meeting and heard about embedded systems. Then it became a concern.

Most of the errors were on the business side and they were ahead of us in looking, and repairing them. They found a problem with the payroll system, it was fixed in early 1999 but we had to keep track of our vacation days for a couple of months into 2000 just to be sure.

We also found a problem with our Digital Fault Recorders. This was after the person in charge of them initially snickered at the idea of Y2K. The information (i.e. currents and voltages) was fine but the time stamp went strange and the time shown was hours ahead of the real time. It was fixed well before the roll over.

We had an interesting reversal of the problem with GPS clocks that Dan mentioned. We encountered a failure after the GPS roll over date that was first attributed to the clock not being updated. After a lot of wondering what went wrong we found out it really was a true (not software or programming related) hardware failure. The clock was replaced.

They did have to update the programming of the SCADA master and the RMS master but this was done well before the roll over and no problems were discovered.

I think the most embarrassing moment was when Secretary of Energy Richardson came to visit and a test (in front of the press, etc) didn't work as advertised. It wasn't a Y2K problem (even though it was a Y2K test) but embarrassing none the less.

======================================

FactFinder's Comments:

Like Dan's experience, the Y2K nuclear plant project I was on was not free of bugs (pun intended) and snafu's either. Here are a few:

1. The vendor of some custom PC based portable data acquisition units indicated that the equipment would work ok in y2k, although the date might be wrong. I tested one of the units, and experienced my first and only failure of a device to perform its intended function due to a y2k problem. During the rollover to y2k test, the unit locked up with the words "divide by zero error" on the monitor. This was pretty funny in retrospect, for I have heard from a number of programmers that "divide by zero errors" were really not the kind of bugs that would occur due to y2k. The fix? It was an old DOS version operating system and older PC, the bios couldn't be "flash" upgraded but I fixed it by installing a patch from the PC manufacturer. Even more amusing is that the units had other problems as well due to poor programming practices, including autorebooting at the rollover to ANY new year -by design, and an inability to handle the last day (366th) of the leap year - the system was in a continuous auto-reboot mode at rollover to the 366th day! Talking to the engineers, I discovered that this problem had actually occurred during the rollover from the 1996 leap year while the equipment was being used in to take data in the control room. As many engineers familiar with computerized equipment ("embedded systems") know, its not uncommon for some of these devices to not process leap year properly, but almost always the dates are insignificant and the functions unimpaired. This "custom" equipment was an exception, since date functionality was an important part of the devices function and the programming deficient. But guess what - by adding the patch files, initiating administrative controls, this equipment was still perfectly acceptable for use except on the days in question. Also to keep this in perspective, I would say better than 95% of vendors gave us accurate y2k information concerning their products. FYI, this being the worst y2k bug that was found at our nuclear plant, I had great hope that mankind would survive y2k after this, the experts notwithstanding ;)

2. Most embarrassing moment - returning to a nuclear plant I had worked at for many years, and belonging to one of the utilities considered a "leader" in their y2k program, I was horrified to see them bringing in porta-johns a few days before the rollover, knowing how the public could possibly perceive it. The plant was confident of its "Y2k readiness, but the contingency planning "program" required this, and this was a good example of "going through the motions" by the planners because the "program" required us to "be ready for anything," no matter how implausible - such were the results of the y2k hype. It was considered pretty funny by the plant personnel (especially to the engineers involved in y2k). The porta-johns were quickly sent away after the rollover.

===============================================

Discussion Question: What do you think was the biggest Y2k screwup? For those of you who worked on Y2k, what screwups can you share?

-- Dan the Power Man (dgman19938@aol.com), June 29, 2000

Bold off

-- Dan (dgman19938@aol.com), June 29, 2000.

What a *riot*. All day long and not one comment from the Doomzies and Magpies here or on E=Z.

Conclusion: those remaining can't even read the post above much less "analyze it". Their "cliches" from the Y2k Kult Mantras don't "apply".

-- cpr (buytexas@swbell.net), June 29, 2000.

Dan:

We work in different worlds. I am not a computer guru [software or hard]; I had to deal with Windows, Mac and Unix. I did virtually nothing. I looked at the information provided and decided nothing was required. I didn't add any of the patches. Everything still works [well, to qualify, as well as it did before; did get some date stamp problems]. The biggest problem that I saw was in mainframes. It was caused by the change over to new software in 1999. All of the bugs that come with new software. I'm not sure how much of that change-over was necessary. Well, who knows. It all works now [sort-of; as usual]. All in all, it all still works as well as 1998.

No date related problems with the infrastructure that I detected.

Best wishes,,

-- Z1X4Y7 (Z1X4Y7@aol.com), June 29, 2000.

I want to take this opportunity to thank you guys for a job well done on this series. I am pleased with the cooperation shown by members of this forum-it shows what we can do constructively when we ban together. I am amused by the everyday conflict lately and it is entertaing, but I am more impressed by a spirit of cooperation.

I just wish I had met you guys before the rollover. Godspeed.

-- FutureShock (gray@matter.think), June 30, 2000.

What a riot! Not a post all day.

Conclusion: those who CAN read have left because of cpr's mentally ill rantings.

-- Bye (seey@later.com), June 30, 2000.

Insiders, thank you for this series and for your work in the field.

cpr, it is rumored that the International Olympic Committee is considering the introduction of Cantankerousness as a demonstration event. So don't cheer up!

-- David L (bumpkin@dnet.net), June 30, 2000.

There has GOT to be a failure to communicate here somewhere. THIS is the worst of the worst?!?

Somehow, the word didn't get out very well. Granted, the vast majority of people paid little or no attention to y2k to begin with. But most of what I saw didn't communicate this perspective at all. We saw budgets and schedules, most of which were sizeable and some of which slipped. We saw the usual (and endlessly repeated) inability to guarantee power at any time. We saw an occasional mention that problems weren't being found, BUT that the search had a long way to go, there were many tests remaining, blah blah. And we saw guarantees from anonymous "insiders" that the likes of Cinergy had NO HOPE.

I always paid attention to TB2K because I knew any and all bad news, however heavily spun, would show up there. The task was to discount the spin and see what (if anything) remained. And then counterbalance that with what the public was being told generally (also discounted for spin). Power was the Big One, and didn't come through this process very accurately.

Your series makes it sound as though, if the entire industry had done absolutely NOTHING, some areas *might* have had some screwed up bills one time. And that's IT! If that's true, the industry was excessively cautious in their pronouncements.

Dan, would you like a great deal on some oil lamps and lamp oil? They're very romantic. I can sell you LOTS of romance, you wouldn't believe. Cheap, too.

-- Flint (flintc@mindspring.com), June 30, 2000.

Future Shock and David L.: You are very welcome. I was lucky that the other guys were happy to share their thoughts as well.

Flint: I'm sure some of your comments are tongue in cheek, but here's my response:

The intent of this chapter was to demonstrate that Y2k was as much a human challenge as it was a technical one--people manage to screw things up no matter how trivial. We did have one problem that resulted in a major software program being shut down for a few hours on March 1 (yep, we got bit with Leap Day, of all things!), but it was transparent to customers.

Regarding guarantees, we were strictly forbidden by the legal folks from using the "g" word, simply because of the liability it could cause. Consider this: The other day we had an outage that was about ten times the size it was supposed to be because of a piece of equipment that failed. The original cause was a non-utility person digging into a power line. All we generally share with the public is the type of outage, and when we expect to have customers back in service. If we say anything about "equipment malfunction" or any other term suggesting that we were at fault, we are inundated with claims from customers wanting compensation. Ultimately we don't have to pay claims unless there was some kind of policy violation or gross negligence, but the costs of dealing with a bunch of claims can be quite high. Thus we have to be very careful about what we say publically. This goes against my tendency to just share the facts.

From a technical perspective, if utilities had done absolutely nothing to prepare for Y2k, there might have been a few problems here and there, but the average customer probably wouldn't have noticed. But from a people perspective, we had to take precautions to demonstrate to the public that we would make sure nothing problematic would occur.

Thanks for the offer on the oil lamps; I'm sure my wife would like one, but I'll pass.

-- Dan the Power Man (dgman19938@aol.com), June 30, 2000.

Dan (and everyone else who contributed intelligently to these power threads):

Thank you for your time, this stuff is very informative. I know myself and others have appreciated your efforts here.

PS - would you object to my compiling these chapters elsewhere (with links back to the originals) for those who want to read them all in a row? [I was thinking of putting them on the deleted forum or Old Timebomb2000]

-- lurky lurker (nobody@im.portant), July 03, 2000.

Lurker: You are welcome. And it's fine with me for you to compile the chapters, as long as you keep the entire chapter text intact.

-- Dan the Power Man (dgman19938@aol.com), July 03, 2000.

I was going to cut the responses to the chapters, instead run a link back to the original. The only thing I would do is "clean up" the HTML, if it happened to need it (unbold this thread for example)

-- Lurky (cut@nd.paste.guy), July 03, 2000.