Factfinder spent some time trying to sort out the information on the BC Hydro website. I have been through the website with a fine tooth comb and even before they had a website I was exchanging emails with them. (One of the early ones came from the project leader himself.}

BC Hydro is being transparent. We can see pretty much exactly what they did, when they did it, and what they found. While I think this is very good news for me, I do not think it is very good news for most other people.

I think we should spend a lot more time looking at the successful companies if we want to understand the broader Y2k picture. We can learn things.

The first thing I learned from BC Hydro is that the entire "percentage of embedded systems" argument is bogus. I think it is fairly easy to reconcile Hydro's 7% number to NERC's 3% number for embedded systems. Generally speaking, BC Hydro has fewer embedded systems overall because they are Hydro, but the embedded systems that cause most of the problems are common to all kinds of generating and distribution facilities.

Saskatchewan Power had twice the number of embedded systems, and a failure rate of 5%. If neither BC Hydro or Saskatchewan Power had fixed a damned thing, Sask Power would have had more embedded systems failures!

Lesson: On January 1st, the percentage of embedded systems that fail is irrelevant. The only thing that will matter is the absolute number of failures. Rick Cowles points out that the prominence of omissions in the NERC report(s). Here is an excellent example. How many failures does 3% represent in a typical power company? That is the number that will matter. The second important thing I learned is that there is an advantage in Canada because our power companies (and other key organizations) tend to be integrated. The supply chain is very short relative to a US power consumer. BC Hydro is a "Crown corporation" which means that it is a private company that is owned by the Crown, the government.

They generate power and they deliver it to me and they come out and read my meter. One company, one successful Y2k project and I get power. In most of the U.S. consumers depend on several companies involved in different aspects of the problem.

Lesson: Canada is probably far ahead of the US in remediation in power, and probably other areas as well. I think Joe Boivan is right when he says, "Canada is leading the race, and we're losing.

The third thing I learned was that it is very hard to complete a successful Y2k project in a large utility.

They started planning in 1994, taking more than a year to decide how to address it. In 1996, they began remediating business systems and they started on the embedded side in 1997.

They started with a complete inventory of all systems. They had more than a quarter million of them, but the vast majority were purely analog. They ended up with 8,000 systems that were digital, systems that could have the bug.

(Note: These are 8,000 distinct systems. For example EMS-SCADA software was replaced in numerous places in the province. That is one system. If there are 1,000 ABC widgets out there, that is two systems. And so on until you hit 8,000.)

They divided the 8,000 systems into one of three categories before they were assessed. High, Medium and Low impact. If all a system did was display a date, the failure would be low impact. The "high impact" category has a definition that is somewhat more stringent than the NERC definition of mission critical. Basically it came down to having a safe system that will deliver power to the customer indefinitely.

They were left with 3300 high impact - or mission critical - devices between the dam and the wall socket. They coordinated with vendors but they did not trust. They tested each and every one of them. (They did do type testing, doing a sample of ABC widgets. If all in the sample passed, all 1,000 widgets passed.)

At the end of the day 223 high impact devices were remediated. While Hydro does not specifically say what kind of failures would occur, the way they defined the problem tells us that these devices would have caused serious problems if they had not been repaired.

(Note again that Hydro has not told us how many different pieces of equipment had to be replaced. Remember that one of those systems was EMS-SCADA replaced in X number of locations.)

They completed this remediation on June 4th, 1999. They are continuing to test contingency plans and work on the medium and low impact systems. (Some of the contingency plan testing has been interesting. Can Hydro get power to Vancouver Island if the grid collapses? Apparently so.)

Lesson: It seems to me that this is the way a Y2k project is supposed to be run. I can look at that process - as a layperson - and decide whoever is running that project seems to know what he is doing.

On the other hand I hear companies that started their Y2k projects a year ago, and some of them are saying that it is a piece of cake, and they are not finding anything important anyway. Do they know what they are doing as well?

BC Hydro started a long time ago, they did it right and they are proud - justifiably - of the fact that they are finished. Like everything else in life, there will be a Bell Curve for Y2k projects. If an "A" student like BC Hydro finished in June, what is the real pass mark? What is the average grade? What do we make about the inconsistencies between what Hydro says, and what NERC implies?

Do I believe BC Hydro or NERC? Was BC Hydro wrong to treat this like a very big deal, or are the cavalier types the ones to worry about? I think a lot of my power company these days, so perhaps I am biased, but I am very glad my power company took this issue very seriously. I wish I could read the same thing into most of the media reports.

My guy is taking this job very seriously and if he says they will get power to Vancouver Island even if the grid collapses, I believe him. I say "Yay, BC Hydro!"

Lesson: Transparency engenders trust. BC Hydro is a good news story for me, but I don't think it is a good news story for us.

Tom

-- Anonymous, August 11, 1999

Answers

Here is an excellent example. How many failures does 3% represent in a typical power company? That is the number that will matter.

Yep. That's what bothers me about reports like NERC's and innumerable press releases: they put out low percentage figures without saying (1) what they're talking about in absolute numbers and (2) what they are talking about in relative importance. And they tell us, over and over again, "oh, most of the problems are things like wrong dates on reports and logs". Phooey. As if I didn't know that without them telling me. As if I didn't know that the key word is most.

Transparency engenders trust. Yep, again. And pseudonymous posters don't understand why they generate as much suspicion as satisfaction. If I were them, I would cringe every time Malcolm Taylor or James Prosser posts here, using real names, providing place names, and lots of details. What do we get from the USA? "Fact Finder" and "Dan the Power Man". Well, don't that just instill confidence? YeeHaa.

I had some private e-mail exchanges with one of the several pseudonymous posters who no longer shows up here. He was obviously quite genuinely afraid that he would get fired if his employers found out that he was coming here and posting good news about the electrical utilities. Well, don't that just instill confidence? YeeHaa.

-- Anonymous, August 11, 1999

Lane, I have seen two types of people involved in Y2k - those that fix it, and those that hype it and whine to those who are fixing it "you're not doing enough! You'll never get it done. too much code, too much code!!!! Were doomed!!!!!!" Guess which category you fit in.

Regards,

-- Anonymous, August 11, 1999

ff,

If the above is an example of the sort of facts you find you should keep them to yourself.

Stev

-- Anonymous, August 11, 1999

FactFinder, that's your big problem. You have some depth, but no breadth. Only two types of people involved with y2k? That is about the stupidist thing you have ever said on this site. Here you have Tom Benjamin, posting an excellent, well thought out article, that you as a polyana should be hooraying over - one more fixed,

Lane Core points out some excellent concerns, which I support. Tom Benjamin makes me want to move to Vancouver, even if it is so damn cold (at least for a Floridian/Texan). People from Minnesota would have their shorts on all winter in Vancouver.

If you have any "Facts" to challenge this good post, lets hear some "Facts" FactFinder.

xBob

-- Anonymous, August 12, 1999

xBob wrote:

Tom Benjamin makes me want to move to Vancouver, even if it is so damn cold (at least for a Floridian/Texan). People from Minnesota would have their shorts on all winter in Vancouver.

Vancouver may be cold for a Texan, but next to Minnesota it is California. Think Seattle. The rainforest. The coast of British Columbia is one of the few parts of Canada that gets very little snow or freezing temperatures.

The comments about moving are interesting and something that I had not thought of before. I expected this to be a stay at home Christmas, but now I wonder whether a person living in say, Toronto, might decide to visit relatives in Vancouver this Christmas because of BC Hydro.

"Hey everybody! It's time to decide where we will hold the Christmas family get together this year. Everyone get out the compliance statements from their power company. What does Ontario Hydro say? Forget Toronto. New Brunswick looks good... New Brunswick or BC?"

Tom

-- Anonymous, August 12, 1999

Maybe you might want to come down here to East Texas. It is warm, and Entergy, our whiz bang power company, who didn't seem to knoow a single thing about y2k, it took me 12 long distance calls last summer to find someone who knew what it meant, has it fixed already. They are miracle workers.

-- Anonymous, August 12, 1999

Looks like I hit the bull's eye.

-- Anonymous, August 12, 1999

Factfinder wrote...

I have seen two types of people involved in Y2k - those that fix it, and those that hype it and whine to those who are fixing it "you're not doing enough! You'll never get it done. too much code, too much code!!!! Were doomed!!!!!!"

Just call me and Lane whining hypsters. If the shoe fits, I'll wear it. Except for the "We're doomed" part. I never say that unless you think having to endure extraordinarily bad times means we are doomed.

I'm not writing to the people who are actually trying to fix the problem. I hope they are too busy to read any of my stuff. I write for other people. My message is "Aah-oooga!! Aah-oooga!! Y2k is a very big deal! Aah-ooga!! Battle Stations! They ain't gonna fix it all, not by a long shot!! Aah-ooga!! Gather your patience and your courage and your rice and brace yourself!! Aah-ooga!! We can face this! We can make the impossibly hard a little easier for ourselves! Aah-oooga!!"

A lot of people think I am a crazy whiner, and I'll happily plead guilty to that, too, since crazy is now defined as being able to add (Y)2(k) and (Y)2(k) and come up with (Y)4(k).

"Aah-ooga!! Crazy Tom is going to go Aah-oooga!! Aaah-ooga!! right up until rollover. Aah-ooga!! Ignore me at your peril! Aah-ooga!! To the lifeboats!! Get your millennium madness alarm bells and join the party. Aah-ooga!! Dive! Dive! Aah-ooga!!"

I've got a bumper sticker that reads "I'm a whining hypster and propud of it. Aah-ooga!!"

Tom

-- Anonymous, August 12, 1999

Tom, I thought that you brought up a very interesting topic in BC Hydro, that's why I took the time to take a look at the y2k information on the www.bchydro.com website. I agree that BC Hydro appears to be doing a good job on y2k based on the details they provide regarding on their website. Based on industry information I have access to, I know that their are a number of utilities are doing an excellent job on y2k, unfortunately they don't all place this information on their website. This is most unfortunate - providing the details of the y2k problems found and fixed and project details would go a long way in allowing the public to see factual information regarding y2k, something that is sorely missing in the bulk of y2k hype posted on the internet.

I personally find BC hydro's y2k information very interesting, since as we have discussed, the information provided infers that the 223 "high impact" devices that were remediated may have been more than just minor date errors. If this interpretation is correct, then this is by far the largest number of "serious" y2k bugs I have seen from one company and way outside of the findings I have found in the US industry databases I have access to. I do not dismiss this out of hand though, since I have seen a previous reference to problems at Ontario Hydro that could have affected power generation (a quote form a company official in a news article, unfortunately I can't find a link to it).

To put this in perspective, I have searched hundreds of industry documents and performed searches for y2k in industry databases, and have been able to come up with only a few cases of y2k bugs that appeared to have the capability to affect power production - and only one of them came from the US (and I cannot even say that this one is "proven", but it did come from an industry source). If anyone is interested, I can put together a summary of my findings, although I will not be able to provide detailed or confidential information.

The nature of y2k in embedded systems is never a "sure thing", since it only takes one (or more) manufacturers of higher level systems to design a system in such a way that date functionality can cause failure to have an impact. But from the best "facts" that I have access too, this just hasn't been the case for the US. Regarding the 2-3% "failure" rates typically reported, these figures refer to failures to process DATES that are tested during Y2k testing, NOT to failures of devices/software to perform their function. "Hard failures" are very rare in the data - there are only a few examples I could site, and all but one were not even installed plant equipment that could impact power production.

Rick often points out that one of the bigger risks of y2k is in plant computer monitoring systems, and I have agreed with this since this is a high level system. I certainly would not just "assume" that since a number of these systems have been remediated (and others are scheduled for remediaton during outages) due to y2k bugs that tthey would have had hard failures (crashed), but on the otherhand without the facts from the utilities involved I wouldn't rule it out.

Which brings me back to BC Hydro - the information presented indicates a potential that this utilitiy has identified devices with possible significant y2k failure modes. This is worth knowing more about. I sent an email to them asking for a clarification of their numbers and terminology, and specifically asked if any or all of the 223 devices had y2k bugs severe enough to impact power generation or distribution. If I receive a reply, I will post it if I get permission from them (I asked for that as well).

As I have pointed out on several occassions, I never claim to have all the facts. I just have a desire cut through the hype and FIND the facts.

Regards,

-- Anonymous, August 12, 1999

Factfinder wrote:

Based on industry information I have access to, I know that their are a number of utilities are doing an excellent job on y2k, unfortunately they don't all place this information on their website.

I am sure this is true. On the other hand it means that management in those companies is not being very smart. It is not your responsibility to counter the "hype" on the internet. It is the responsibility of the power company. If they are doing a lousy job with the public relations part of their Y2k project - and public relations is part of the Y2k project - why should they get the benefit of the doubt on the rest of their project?

I personally find BC Hydro's y2k information very interesting, since as we have discussed, the information provided infers that the 223 "high impact" devices that were remediated may have been more than just minor date errors. If this interpretation is correct, then this is by far the largest number of "serious" y2k bugs I have seen from one company and way outside of the findings I have found in the US industry databases I have access to.

I think this is a function of size. In Canada our power companies are huge. They are Crown Corporations, companies that are owned by the state. We do not have 8,000 power companies in Canada. We have 14. (Don't hold me to that number. The point is it is a very small number of very large companies. Relative to the population, it is like the US had 200 compnies in total, not 8,000. The power industry is structured differently.)

To put this in perspective, I have searched hundreds of industry documents and performed searches for y2k in industry databases, and have been able to come up with only a few cases of y2k bugs that appeared to have the capability to affect power production - and only one of them came from the US (and I cannot even say that this one is "proven", but it did come from an industry source).

I think Hydro would probably nearly agree with you. The generation of power is the easier part of the remediation, particularly for Hydro plants. The stickier problems are in the transmission and distribution and the telephone systems. What good is power produced if it does not get to my wall socket? How many companies in the NERC survey generate power, transmit it, distribute it and sell it directly to the customer? BC Hydro does.

In the US several companies might own the power as it makes its way from the power plant to the wall socket. BC Hydro had the same number of problems to solve as all of those companies would collectively.

The nature of y2k in embedded systems is never a "sure thing", since it only takes one (or more) manufacturers of higher level systems to design a system in such a way that date functionality can cause failure to have an impact. But from the best "facts" that I have access too, this just hasn't been the case for the US. Regarding the 2-3% "failure" rates typically reported, these figures refer to failures to process DATES that are tested during Y2k testing, NOT to failures of devices/software to perform their function.

This is wishful thinking. If you read NERC's definition of a Y2k ready device, it is a device that is suitable for use in the year 2000. A compliant device is one that handles dates properly. In other words, a device that works perfectly but displays the wrong date could be called Y2k ready and ignored by a power company.

NERC's definition of mission critical includes only devices that could impact on the generation and distribution of power. If what you say is accurate, no power company has any mission critical systems to remediate at all.

The only embedded system Hydro names is EMS-SCADA. I understand it is possible for Hydro to function in some fashion without it, but Hydro did not think it was safe, or they did not believe they could sustain the system without EMS-SCADA. Partly this is a human resources issue. How many power plant engineers know how to do their job without EMS- SCADA? How many extra qualified people would they need if this monitoring systems aren't there to assist?

Plus you are focusing on the narrow generation of power issue again. For the company to function for more than a few weeks, they have to be able to generate the power, they have to transmit it, they have to distribute it, they have record what power they sold and when it was sold and to whom it was sold. They have to generate a bill.

Rick often points out that one of the bigger risks of y2k is in plant computer monitoring systems, and I have agreed with this since this is a high level system.

BC Hydro generated and distributed power long before there were computers. Most of that equipment has not changed. What has changed is the monitoring systems. They make the plants more efficient in two ways. The same number of plants can serve more people with more efficient load balancing and distribution. Second, you need a lot fewer people who are doing different jobs. Computers changed the power business liked they changed every other business. Most of the workers know the business with (fairly) reliable computers. They do not know how to operate the business without them.

The really bad things will happen because a minor system fails and combines with another minor system failure to produce a confusing situation for a (probably tired) human being who will make a stupid mistake.

As I have pointed out on several occassions, I never claim to have all the facts. I just have a desire cut through the hype and FIND the facts.

This is fair enough, I suppose, but I don't know what hype you are talking about. I monitor the Y2k situation pretty carefully and I do not see a lot of hype about power. The first NERC report outlined the issues pretty clearly. The Senate Report does too. Cowles and Mills see exactly the same issues and threats.

What hype are you cutting through?

Tom

-- Anonymous, August 13, 1999

What hype are you cutting through?

He must be worried about all those millions and millions and millions of people who think that having an extra can or two of soup will get them safely through Y2K. (See, for instance, Se nators' advice is priceless.)

-- Anonymous, August 13, 1999

Actually, Lane, that letter to Ann Landers is exactly the sort of "hype" or disinformation that has troubled me for a year now. Have on hand a flashlight, some batteries, some soup, etc. This is not new stuff that is being recommended. Probably 90% of the people who read that will say "Whew, I already have that stuff, I'm set." Yet, he goes on to tell you to *research* this thing. Contact a lot of people/businesses you rely on. Keep bank records. Check with your doctor about prescriptions. What? Huh? Why? If all I need is few cans of soup and a flashlight, why go to all that other trouble. I don't get it. Or should I say, They Won't Get It.? And they are not intended to get it. If there is hype, that's where it is.

-- Anonymous, August 13, 1999

Tom wrote: This is wishful thinking. If you read NERC's definition of a Y2k ready device, it is a device that is suitable for use in the year 2000. A compliant device is one that handles dates properly. In other words, a device that works perfectly but displays the wrong date could be called Y2k ready and ignored by a power company.

NERC's definition of mission critical includes only devices that could impact on the generation and distribution of power. If what you say is accurate, no power company has any mission critical systems to remediate at all.

I am familiar with the definition of Y2K Compliant and Ready. The nuclear y2k projects I have been involved in used the definitions from NEI/NUSMG, and they are similar to NERC. You miss my point though - I did not say that there were no bugs in mission critical systems. To repeat: "Regarding the 2-3% "failure" rates typically reported, these figures refer to failures to process DATES that are tested during Y2k testing, NOT to failures of devices/software to perform their function."

The 2-3% "failures" most certainly do include a number of mission critical systems, I have been involved with y2k remediation of several of them myself. For generation and distribution devices that could cause a power outage, the findings of the US electric utility industry are that almost all of the y2k bugs do not prevent the "mission critical" devices from functioning. This isn't "wishful thinking", these are facts from within the industry. As far as business process software that is mission critical, yes there have been some serious y2k problems since dates are often critical (used for calculations, etc.). Business software won't shut the plant down unless the company folds...is THIS what you are trying to say? How many power companies do you expect to fold as a result of Y2K "failures"?

And Tom, you had me ROFLMAO when you asked: "What hype are you cutting through?" You're kidding, right? Umm...you've never seen hype concering Y2K, have you...lol.

Well, here's one example - you just posted it:

"Plus you are focusing on the narrow generation of power issue again. For the company to function for more than a few weeks, they have to be able to generate the power, they have to transmit it, they have to distribute it, they have record what power they sold and when it was sold and to whom it was sold. They have to generate a bill."

Nonsense Tom, this is nothing but the kind of smoke and mirrors Y2K hype I'm taling about. Generating plants AND distribution systems in the US have been found to contain very few y2k bugs severe enough to prevent generation and distribution. And are you seriously implying that you might not get a utility bill in January? lol...rest assured, billing software remediation was completed first...;)

My second example would be almost any y2k article with the following attached "copywrite Lane Core", lol.

Regards,

-- Anonymous, August 14, 1999

Fact finder wrote:

The 2-3% "failures" most certainly do include a number of mission critical systems, I have been involved with y2k remediation of several of them myself. For generation and distribution devices that could cause a power outage, the findings of the US electric utility industry are that almost all of the y2k bugs do not prevent the "mission critical" devices from functioning.

I understood you the first time. Garbage. The argument is inconsistent with the NERC definitions. The failures you describe don't count because those systems are Y2k ready. They are suitable for use in the year 2000. Who cares what date is displayed? One in the plus column on the old NERC survey.

"How about this mission critical system?"

"Will it work?"

"Except for it displays the wrong date, it functions."

"Y2k ready."

"Check."

If every embedded systems failure in a power plant was like this then all of the mission critical systems will function and the plant is Y2k ready. Oops. There are some unremediated plants, aren't there? Conclusion: Not all embedded systems failures are like that. Relatively speaking, a very few mission critical systems stop functioning on rollover, but very few is different than zero.

Here is another clue. The word "almost" in your paragraph. You agree with my conclusion. A very few mission critical systems stop functioning on rollover, but almost is different than all.

Final clue: BC Hydro agree with us too! They have about 250,000 systems involved in the generation and distribution of electricity. About 249,777 of them were Y2k ready and 223 high impact systems were not, at least not before remediation. I would say that is "very few" critical systems will fail and I would say almost all systems are ready. If I wanted to be really misleading I could say I was 99% done before I started couldn't I?

And Tom, you had me ROFLMAO when you asked: "What hype are you cutting through?" You're kidding, right? Umm...you've never seen hype concering Y2K, have you...lol.

Well, here's one example - you just posted it:

I'm the one hyping the issue? You are protecting people from me? The mainstream media is ignoring the issue, but as long as some nobody named Tom Benjamin posts comments about it in a couple of internet forums, the issue is being hyped? Lane Core is hyping this issue? Putting doom and gloom on the lips of every American?

Core and I are pasting CNN on this one, are we? The Happyface is on the verge of collapsing, about to crumble under our relentless onslaught of hype. Our strength must be the strength of 10 million because our hearts are pure and our cause is just.

There would be a panic for sure if you weren't around to cut through the smokescreen of hype produced by me and Lane Core.

Nonsense.

Tom

-- Anonymous, August 14, 1999

It's all very clear now. We have no "Fact Finder" here. Never did. We have a Stephen-Poole wannabe, who categorizes everybody who disagrees with him as a whacko, and then proceeds on a half-witted smear campaign.

Thanks, Fact Finder. I used to think you were an honest man. No more. Thanks very much for the revelations of these past few days.

-- Anonymous, August 14, 1999

I don't think anyone here realizes the shots those of us in the industry take from those in the forum who have a different opinion. Many of you have no problems making posts that in some way belittle others, but you sure don't accept having such posts made about you. You dish it out, but boy, as shown above you certainly dont want to take it. It was the same way for cl when he posted here. I think that Dan the Power Man has exhibited very good self control here, yet even he gets your usual put-downs.

As far as who I "wannabe", well I would like to be me, exept maybe with a little more self control at times, lol. Poole? I sent him an email one day months ago concerning what I saw were unecessarily personal attacks of Rick Cowles on his website- he posted the email on his website. I know things sometimes get a little testy and personal in posts, but that's often in the heat of battle, I don't like to see it in a website (good natured jabs, fine, but not going for the kill). Now having said that, I don't think that you, or I, or anyone else who has EVER belittled, slammed, or attacked someone else has the right to think we are somehow superior to anyone else.

Lane, I thought you could handle a jab, you certainly can throw the punches, lol. Look  I took a shot at your articles, but it was just a jab with a lol attached. I think you know I respect the research and thought in your articles enough to discuss some of them with you even when I disagree with some of your interpreations and conclusions, since we have done this. As you may remember, in one post I even agreed with you about the September 9, 1999 as a bogus "critical" date.

And finally, Lane, I want to point out that it was you who took the first cheap shot in this thread, something you have done on a number of occasions in this forum  I can provide the links to the posts if you need, so you can put that bit of self-righteous indication right back in the bag and take it back to the store where you bought it, lol.

Regards,

-- Anonymous, August 15, 1999

Lane wrote: "What do we get from the USA? "Fact Finder" and "Dan the Power Man". Well, don't that just instill confidence? YeeHaa"

Lane - drop an e-mail to Drew Parkhill. He has verified that Dan is a true-blue T&D power man. Dan offered to establish his credentials when the mob at TB2000 were trying to give him the bum's rush. Drew helpfully volunteered to be the "honest broker", and when he was satisfied that Dan actually was who he purported to be even the hardcore types at TB2000 accepted his [Dan's] posts with no sniping.

WRT to FF, maybe if you ask nicely he might agree to do what Dan did.

-- Anonymous, August 16, 1999