Some rambling thoughts about my rollover....

I was the communications center captain for the operations center at the power company I work for...kind of a "straw boss" for the evening. I couldn't get the phone/pager/radio to stop going off, once even in the restroom!

The day started out very early, with New Zealand rolling over. While most were watching the fireworks on TV, I tried to look beyond them to see if the lights stayed on! So far, so good...

Then came some scary news, about the transmission line out in Oregon with "sabotage suspected". Many of us feared this was only the beginning of several of these incidents...

When relaying the above information internally, I kept emphasizing that we were not to share this outside of our company, as per the request of the other power companies. Then, lo and behold, I see it pop up on a public web site..."How did that get there?" I wondered. Over to NERC's site, and there it was. I was frankly very surprised that they made that public so quickly.

As each country continued rolling over came increased optimism for the outcome. The EPRI early warning system provided more good news on specific power devices.

The good outcome resulted in the media pretty much completely ignoring infrastructure items, and focusing more on the fireworks and partying (sigh).

The GMT rollover did prove to create a glitch that had some of our folks scrambling for a while, but the problem got resolved in about an hour.

The actual rollover didn't result in any outages, but a couple of non-critical programs did malfunction. One of them was "Vendor tested and certified". Arrgh! I felt glad that we had tested all mission critical device types ourselves to be sure.

I must confess, a few months ago I thought I'd feel vindicated if there weren't any outages, but now that it has happened, the prevailing feeling is gratitude. I'm grateful it all seems to be working out, and grateful for my co-workers, willing to help out if needed.

It has been one heck of a wild roller coaster ride. Isn't it ironic that what was supposed to be the wildest part of the ride was so very boring, yet I feel satisfaction?

Then by 4am it was time to hit the sack, exhausted but happy it's over.

I'll post my analysis later this week when we find out more about the business systems and other issues...here's to hoping that the rest of the ride is uneventful as well...take care all...

-- Anonymous, January 02, 2000

Answers

Dan

Thanks for the update post.

-- Anonymous, January 03, 2000

Dan,

Well, I owe you an apology, along with FactFinder, CL, Menno, Prosser, and others, so here goes.

I was so darn sure that we would have significant problems in the power and telecoms in at least a number of "under prepared" countries. It didn't happen. Now I will be facing a medical bill, because my jaw dropped early on 01/01/2000 and I'm still having trouble getting it to remain closed. Probably from clenching my teeth so hard for the last 18 months or more. Also, my head keeps moving back and forth from side to side. Probably from being so negative about all your optimistic comments for so long. And now it is a bad habit that I have to get corrected as well. Serves me right.

I can see now that I'm not the only one who drew some lines and then projected that into the future where it went into "tiger county". Guess I need a new French curve in my plotting tools. Oh well. But more importantly, I was really hard on all the (what I called Polly) forecasters here and other places. I just couldn't accept that the electric and phone would work without major incident everywhere. You said it would, but I didn't believe it. You were right and I was wrong.

I don't know what the next days and weeks will bring, but the litmus test for me was the power and telecoms. With so little trouble in that sector, I'm going off the "high concern" mode from here on out. I have used analogies in flying a lot here, so let me offer one more. When we were concerned about turbulence we put on the seat belt sign and made an announcement to the passengers to remain seated please. At some point we had to reevaluate our situation for further probable turbulence and if it looked like we would be in pretty good shape we would turn off the sign and let the folks move around again. That's where I am now with Y2k. Seat belt sign is OFF now. Could we hit some bumps ahead? Sure. That's always a possibility. But if it's not something that we think will knock people off their feet, the sign stays off.

Good job, you guys, both in the work you did and in staying the course with your positive comments in the face of some harsh criticism from me and others. I know you folks are hard working pros. I can deal with some screwed up records (if that happens) as long as I have the electric power and the phone working. And I have had that, without even a momentary interruption. Didn't have to resent even one digital clock during the last 48 hours. Can't argue with that. And can't (and won't) argue with you or the others from here on out. I'm not such an egomaniac that I can't give credit and respect where it is due. From here on out, as we watch the power industry try to deal with deregulation, and as long as this forum stays up, if there is something I want to ask and you give me your opinion about it, I will listen with the sort of ear you deserve to receive. Best to you.

-- Anonymous, January 03, 2000

Dan, great report, thanks! Gordon, what a very nice and humble reply. But sir, you certainly don't owe me an apology, we were all discussing y2k based on the information we had, and skeptism is always healthy, I appreciated our discussions. This was never a who was right/who was wrong issue, but simply one of industry having to deal with a widespread common computer problem, and would we fix it adequately in time to avoid major problems. Fortunately for all of us, it appears that the job was "good enough", but as you say, we have a few days to go. To date however, we should all be pleased that the US (and all the other countries?) apparently had no power outages due to y2k.

I find some of the problems at the nuclear plants a bit suprising, I truly didn't think we would have more than a couple of very minor bugs in the more critical systems. For example, I was quite suprised at the security system problem. We had knowledge, we had time, yet we missed a few of these, and I am not comfortable with that. I fully expected to see minor problems that were missed in non-mission critical systems, but critical ones I had zero tolerance for failures. I am a bit of a perfectionist perhaps, but I would like to have seen few reports for me to be in my personal "comfort zone". So while we had very good success, my opinion is we should have done much better. Dan, I'm interested in your thoughts on this.

Regards,

-- Anonymous, January 03, 2000

George: You are welcome.

Gordon: Thank you very much for your honesty and candor. I think you have done an honorable thing, and I want you to know that I respect that very much. Apology accepted. (Dan electronically shakes Gordon's hand, and we both call it good).

I loved your "jaw dropping" analogy. I think I need some "de-hazing" medication, as I am a bit surprised how well things have gone so far.

FactFinder: Actually, as I suggested above, I expected a few more problems. My initial suspicion is that several other unreported glitches occurred as well. I believe this to be true because of the range of approaches to Y2k that I observed. Although for mission critical items utilities were pretty consistent, for those other items (like monitoring equipment at power plants) there seemed to be a wider range of approaches: Some went strictly with vendor assurances, whereas others did full end-to-end integrated tests. My hope is that all glitches get reported so that all companies can benefit. I'm glad to hear your neck of the woods went pretty well.

-- Anonymous, January 03, 2000

Gordon, can I go to the doctor with you for "dropped jaw" and get a group discount? I'm still mystified at all the systems that are still operating.

FactFinder, thanks very much to you and all your EU buddies who have worked so hard to keep the lights on. I guess that the extra can of beeny-weenies I saved for you is going to be collecting dust, eh?

Like Gordon, I'm grateful. I prepared my head off, and fortunately never had time to can 14 or so frozen turkeys. Other aspects of my life can now get back to normal now. Anybody need any lamp oil? Thanks to all of you--pollies and doomers-- for your encouragement, the mature nature of the discussions here, and for being so open. You're a great bunch of people.

-- Anonymous, January 03, 2000

Dan, Factfinder - congratulations on being right. Something I don't understand though, relates to a question I raised several times in the past. How come, after the NERC (and you guys) had stated that there was no risk of y2k related disruption in the electricity sector, so many governmental agencies in the US claimed so many countries as being at 'moderate to severe' risk of y2k related power failures? How come these guys, the late starters, those who did nothing or next to nothing about remediating their systems, how come they got the same result as you guys, who are now being revered as heroes.

-- Anonymous, January 03, 2000

Hello Chris: I'll answer your first question the same way I did a few months back, based upon what limited experience I have on the subject. The reasons that there were dire warnings about power systems in other countries is that there simply was not sufficient information about their systems and remediation efforts. There were rumors of substandard design practices, use of pirated software, and outages that were a much more common occurrence due to a much more fragile system. I know of one country that flat out refused to give any Y2k information to a major company with a manufacturing plant there. When faced with this kind of uncertainty, the prudent thing to do is to make contingency plans accordingly.

Why did they all sail through so well? I'm not so sure that they did...they might have averted immediate problems, but I'm guessing that they may have many patches and work-arounds to develop. Anyway, one reason they did well is what you've heard from us "insiders": Power system equipment is for the most part date-immune. What equipment does use dates uses it only for time-stamping. And those utilities that did perform their own testing merely confirmed what the manufacturers were saying.

And as far as being "regarded as heroes", I sure don't feel that way. A few co-workers shook my hand and gave their congratulations...my boss wrote an e-mail that basically said "sure, you did a good job for the rollover, just make sure you don't screw anything up between now and leap day." I think the thing today that made me feel pretty good was Gordon's post above.

Actually, I plan to post a short list of Y2k "heroes" from an industry insider's perspective some time in the next few weeks. And no, I'm not on the list ;-).

-- Anonymous, January 04, 2000