Wouldn't the San Francisco outage be considered a Y2K failure?? As I understand it the power outage in San Francisco was caused by a repair/replace team, updating a non-compliant line, shorting out the San Francisco Bay area?? If they were updating for Y2K, IMO, that is a Y2K failure. This also shows what type of problems might be more common during late 1999/early 2000 Y2k repairs. Manic management pressuring exhausted work crews could cause more problems than Y2K! Any comments...

-- Anonymous, February 11, 1999

Answers

Mark, I find it difficult to comment, or even care, without a source for your information.

-- Anonymous, February 11, 1999

If you read the reports on the incident you will find that the outage was caused by workers leaving temporary grounding wires installed. This was an OFU (operator f**k up, in this case worker, but you get the gist of it) not a Y2k failure.

-- Anonymous, February 11, 1999

I did read the reports and they said it was to update for Y2K. Which means it was caused by Y2K. Doesn't matter what the reason (bad parts, software or operator error) an outage occurred due to Y2K work. If a company was doing Y2K remediation on its software, though it was complete, put it into production and the fix deleted all hard drives, that, IMO, would be a Y2K critical failure. The above example is hypothetical. My point is.. if something is being fixed for Y2K and that fix causes problems, wouldn't that count as a Y2K failure??

-- Anonymous, February 11, 1999

The official excuse from PGE was "human error" (i.e. leaving the ground wires connected on start-up or some blather like that). The California PUC did not buy it and immediately launched an investigation. I too thought the blackout was coincident with Y2K testing and repairs that should be going on right about this time. This is just conjecture on my part. If it was Y2K related and the company had admitted it, the revelation would have set off a national panic in my opinion.

What report did you review other than general news media stories?

-- Anonymous, February 12, 1999

Folks, I think that The Important Lesson To Be Learned here is that cascading failures that sweep out far beyond what should have been a very localized one, are completely verified as of just a couple of months ago. Y2K promises to trigger just this sort of thing: multiple, simultaneous "local" failures of various components, with possibly a very far reaching effect.

-- Anonymous, February 12, 1999

Joseph,

From the industry news article that came over my company internet, closing on a safety ground WAS the cause of the SF outage. Background: When working or maintaining high voltage eqpt, all three phases are grounded where they connect to the line. That way, the workers are inside the grounds, if someone accidentally energizes the line from the opposite end (next substation down the line), the current will be "stopped" at the grounds and the workers will be safe.

SOMETIMES, people forget, and the grounds don't get removed. This is a very serious error and quite a spectacular show. Hope they were wearing brown trousers. Unfortunately, this introduces what is referred to as a "bolted 3 phase fault".

A bolted 3 phase fault is the most severe fault a system can endure. The engineering problem this particular fault presents is even mor difficult because 1. It was "close in", or right on top of the circuit breaker. 2. It occurred on initial energization - the protective devices did not have any pre-fault voltages to polarize them (the open circuit breaker and the grounds caused this). This presents a special case that is solved with protective schemes known as ZVT or Zero Voltage Tripping. It is difficult because the information available to the protection is less than normally avail., and the severity of a bolted three phase fault requires quick action. (Its sort of equivalent to police busting down a door and instantly having to decide if the occupants are armed bad guys (shoot), or hostages).

At any rate, standard performance criteria requires several levels of contingencies to accomodated without cascading outages. A three phase fault at the location of the SF incident should have been one. I am past my knowledge of what actually happened there - so here on is speculation. I believe that another failure or human error had to occur in conjunction with energizing onto grounds. This is what we refer to as a "brain fart", and is quite rare.

Three phase faults are EXTREMELY rare. I doubt ANYONE will be allowed to be doing maintenance on critical dates - so safety grounds will not be an issue. Any fault would have to occur precisely at the rollover - quite a coincidence, too remote to discuss and not be laughed at. IF a fault would occur, the protection would have lots of pre-fault polarizing voltage and would be set up with it's best protective algorithms enabled.

Bottom line, I don't think the SF outage is relevant to Y2K. In general, for cascading outages to occur, there would have to be a SEVERE shortage of generation to feed the load on the grid. If the generation boys do their stuff, or if the generation plant problems are sporadic (as some predict - but I don't see much objective data to base the prediction upon), I think that standard utility lines of defense should prevent a widespread cascading outage. (see my comments on the thread "what is a rolling blackout".

-- Anonymous, February 13, 1999

Mark,

You said "I did read the reports and they said it was to update for Y2K. Which means it was caused by Y2K. Doesn't matter what the reason (bad parts, software or operator error) an outage occurred due to Y2K work"

If I am driving to a substation to perform a Y2K conformance test, the lid comes off my coffee, burns my leg causing me to run a red light, forcing a truck off the road and into a utility pole, leading to an extended outage. Would this be a Y2K failure??? (grin)

On the lighter side, A utility remediator tasked reported to management that the Y2K effort was 100% complete. Every company document and line of code had been been been reviewd and edited changing all the Y's to K's. The remediator also commented that while they were at it they noticed what appeared to be a problem with 00 being misinterpreted in the year fields - someone might want to look into that also.

-- Anonymous, February 15, 1999

I just wanted to say, that I can understand the argument that Y2K is partly to blame for the outage if it is true that the grounding wire was set for the purpose of applying a Y2K compliant component. In this case, if it wasn't for Y2K then the ground wire would not have been placed. This is a common problem in computer programming. The more "fixing" you do to a system, the higher the probability that you will cause an error.

-- Anonymous, February 16, 1999

Troy,

Do you have information that would rule out the possibility that the equipment was not SCHEDULED for maintenance and the Y2K check was scheduled concurrently? Grounds are necessary for ALL maintenance work on primary equipment. See comments by Art in the rolling outage thread. Excellent info.

-- Anonymous, February 16, 1999

It appears the San Francisco outage in December was caused by a combination of human error, communications problems, and failure of a protective system. No news article I could find stated why the new transformers were being installed.

From the kpix.com news archives:

The new report blames a much-maligned construction crew for failing to remove protective grounds at the substation before allowing power to come back on, but also points to a second mistake when a transmission operator failed to activate ``protective relays'' that could have tripped circuit breakers when the lines were re-energized. ``Had either of these failures not occurred,'' the report concluded, ``there would have been no outage.'' As it happened, within 11 seconds, two power plants in San Francisco also shut down. In their summary of the report, similar to one sent to the California Public Utilities Commission today, investigators mentioned several types of ``missed opportunity'' that should be avoided by better checking in the future. They noted that one person should be designated to monitor the removal of the protective grounds, noting how many were installed and how many removed. ``PG&E's transmission operators failed to follow established procedures requiring that all steps in a switching operation be separately entered in the switching log,'' the report noted, adding that several training sessions already have been conducted. The report also criticized internal communications between San Mateo and two other regional PG&E centers as inadequate on that day. ``Some confusion occurred in these communications,'' it said. PG&E vowed to connect San Francisco's power substation modems to batteries for backup power and add a second layer of review for switching operations.

From PG&E's web site:

Triggering event

1. A construction crew working on the installation of new transformers at San Mateo substation failed to properly remove temporary grounds. These grounds are installed to protect employees while they are working on de-energized equipment. We believe that several safety procedures and precautions were bypassed.

2. At 8:15, according to a detailed startup switching checklist, we believe the operator closed a breaker which connects the grounded fault into the electrical grid. There is a protective system installed on each bus to detect any abnormal flow of electricity across the bus structure and instantaneously (in 1/10 of a second) separate the system from the fault.

3. We know that this protective scheme did not operate. We do not know if this failure was due to continuing human error, failure to follow work procedures or electrical/mechanical failure. We are testing each component separately to determine any problem with the equipment. This is a focal point of the continuing investigation and more testing will be done.

4. When the grounded bus was energized and the protective scheme did not operate, it took 4/10s of a second to isolate the ground instead of only 1/10th of a second. That may not sound like much, but in the world of nanosecond high voltage electrical flow, that is almost an eternity.

-- Anonymous, February 16, 1999

At this point, I would give the chance that it was a Y2K induced about a 10% chance. I would say at this point that it was most likely not Y2K related at all. My reasoning is that transformers, as far as I know, don't present a Y2K problem since they are basically full of wire not chips. It doesn't make sense to me that this would have been a Y2K update of any kind. I threw in the 10% just because I still don't have enough information to prove that they weren't adding the transformers for Y2K reasons.

-- Anonymous, February 16, 1999

Art,

Looks like you nailed it in your guess of what caused the extended outage! Good reporting Bonnie, and great analysis Art!

-- Anonymous, February 16, 1999