August 21, GPS and the Grid

By Tim Castleman

Copyright 1999, all rights reserved

Like the ripples on a pond when a stone is tossed in, Midnight, August 21, 1999 signals the beginning of the instability of the grid. That is when the GPS satellite clocks will reset to zero, and certain devices that rely on that signal will fail. The actual quantity of devices that will fail is unknown. Receivers manufactured before 1994 are especially prone to failure. All utilities employ a very precise portion of that timing signal as a source for an array of data recording and control systems. Timing is everything in the utility business. Especially power generation and transmission.

Large amounts of bulk power are shipped around via a massive web of interconnected high voltage wires. Variations are measured in every imaginable way, but frequency is the most watched. Small moves in frequency can develop into big troubles, tripping automatic devices and overloading things here and there. That's how brownouts and blackouts happen.

A sophisticated computer system keeps track of all this, adjusts things, records things, accomplishing major tasks of data management that would require hundreds of personnel manning millions of switches, which would be far more error prone than the computer. These systems are referred to as SCADA and EMS.

So, when the timing signal data starts to get a little screwy, things will begin to get a bit unstable. Not all at once, it will be a cumulative effect. Small errors will enter the stream of information constantly flowing from facility to facility, via satellites (more about these later), phone wires, radio signals etc.

These sophisticated systems will initially make minor adjustments automatically, and certain operations always require operator intervention and or approval. These minor adjustments are recorded and logged, used for future forecasting and so forth. So now the slight errors have entered the database. Once there, the corruption, and eventual failure of the system has begun.

As larger scale testing gets underway during August and September, the potential exists, and in fact has happened, of system failures due to the testing. Human error, Back up systems not ready, whatever. More corrupt data enters the stream.

October marks the new final deadline for 100% readiness. Even that has become a farce. Gone is the term "Compliant", no one seriously thinks that could happen, and they are right. There is no way to fully remediate and test the whole system. It simply cannot be done, no matter how much time we had.

Instead we have "Ready", which is a watered down version of "Compliant". In short, it says that pretty much any method goes, windowing, date interception, date expansion, TSR's (an old fashioned DOS thing that resides in memory, and caused all sorts of havoc then), whatever. Patch it. It's ok. Testing will reveal any problems, and the emergency response teams will solve them. Becoming "Y2K Ready" simply means you are working on it, and have a team of experts on call to fix problems AS THEY COME UP.

No one knows for sure how many systems will fail, or when. A certain percentage of failures, added up over time will lead to spot outages. For example, the NERC database reports that out of 193 EASTERN Interconnect companies, 181 expect a failure. This failure could be something as simple as a fire alarm going off, or security system going haywire, or, it could happen in a SCADA /EMS system. Of course the latter is a worst case scenario, right? Of course.

Here is a list of "Exceptions" allowed by the NERC for utilities reporting,

Non Nuclear exceptions:

Emmision monitoring Generator controls SCADA/EMS Communications Customer support systems Justifications:

Upgrades and Testing Vendor availability So, added to the watered down, "Y2K Ready" requirement, there are "Exceptions" allowed if the reason for non-readiness is due to Upgrades and Testing OR Vendor availability.

A company can claim readiness, even if they are not ready, simply by stating that the delay is due to Upgrades and Testing, to any of those systems listed above, or, they are waiting for a vendor to respond in some way. Most are waiting on the vendors.

I am using data from the July 10 EASTERN Interconnection report to the NERC for this article. In the summary of the data collected, the average completion status is given as 99%. That means on average, utilities in the eastern interconnect did not make the July 1 deadline, even the greatly relaxed "Ready" deadline!

Names are not given in the copy I have. Each utility is identified by a number only.

Back to the satellites. There are 24 for the GPS alone.

There are thousands more for all kinds of communication. On board those satellites - the same technology we are wrestling with here, only worse. It is not possible to physically access them at all. This is the equipment called for in all contingency plan details made public to date, to provide back-up communications for all utility companies, with a hot line directly to the DOE.

In addition to the obvious rollover issues, solar flare activity is increasing now and is expected to continue to increase culminating with an 11 year high peak in December, 1999 and January, 2000.

These flares generate enormous amounts of electromagnetic energy. The last peak was in 1989 and was responsible for the grid outage in Quebec. Satellites are at such great risk that evasive maneuvers are planned during these times, things like turning some part of it away from the sun, putting exposed units in "Sleep" mode etc.

The President issued an Executive Order July 14, 1999, which states in part:;

"The NIAC will report to the President through the Assistant to the President for National Security Affairs, who shall assure appropriate coordination with the Assistant to the President for Economic Policy."

NIAC = NATIONAL INFRASTRUCTURE ASSURANCE COUNCIL.

http://www.pub.whitehouse.gov/uri-res/I2R?urn:pdi://oma.eop.gov.us/1999/7/15/10.text.1

National Security Affairs is involved now. For the next two years, three if needed.

To summarize:

About half of the companies will not be testing the work they are doing. About the same number will not make this information available to the public.

As a whole, the industry did not meet the July 1 deadline The "Exceptions" list includes the most critical systems for power generation and transmission.

In May, 1999, 42% of Public utilities and 18% of Co-ops knew they would not be ready by the July 1 deadline, and so did the NERC Y2K Coordinator, Gerry Cauley. When asked if they had completed testing of SCADA/EMS systems, 17 said N/A, 90 said yes, and 50 said NO. Also, 44 had not completed contingency plans (again missing the July 1 deadline). Of 193 EASTERN Interconnect companies responding, 181 expect failures Nothing is mentioned about the GPS in either of these reports

Executive Order July 14 would not be necessary for a system as close to "Ready" as claimed. These are disturbing facts. Even more disturbing is the ongoing cover up and media campaign to keep these details secret. The general public does not get much of a chance to make prudent preparations. One of the slides uses this statement as a title:

"Defense in Depth Strategy: Emergency Preparedness is Important Step"

So, as concerns the NERC and utility companies, Emergency preparedness is an important step. But when the average Joe Public says that, he is branded as a lunatic fringe alarmist.

Ok, this lunatic fringe alarmist has uncovered hard evidence of the cover up. I have several Powerpoint presentations intended for internal use at the NERC and almost 8 megabytes of Excel Spreadsheet data to prove it.

The bottom line is that no one knows for sure how much will fail, just that it will.

Sources: Excel Spreadsheet dated July 10, 1999 - Summaries of June 1999 reports by utilities to NERC "An Industry Status Report and Plan", May 12, 1999 by Gerry Cauley, Y2K Program coordinator, NERC (Powerpoint presentation) Salt River Project Year 2000 information, "Technology newsletter", February, 1999

-- Anonymous, July 31, 1999

Answers

It all sounds, oh, so very reasonable. But, when dealing with physical realities, "reasonable" doesn't mean "true". Often, what seemed reasonable beforehand turns out to have been way off, and what seemed unreasonable or even un-thinkable turns out to have been right on.

Like the ripples on a pond when a stone is tossed in, Midnight, August 21, 1999 signals the beginning of the instability of the grid.

We shall see. Or not.

-- Anonymous, July 31, 1999

I love spreadsheets. I know, that sounds dumb, but as an analyst, I love to cut up numbers and put them into perspective to gain knowledge. It's fun. However in this case, the numbers aren't really that fun. In fact, they are a little disheartening. Thanks to Bonnie for that FTP link in prev. post and to this poster for jogging my sleep deprived mind to look at my downloaded files.

I went and took the June 99 summaries spreadsheet (special shout out to my boys at NERC for putting this together so that we could use it to nail you down ;) and started playing the percentage game. You know the one Washington and Koskinem use all the time to keep the Sheeple in the dark.

Anyway here's some initial figures. Rest assured I will be doing more digging to look at some of the other sections of the report, but here's a taste....

6. How will your organization establish Y2K readiness? For EMS/SCADA systems? (this is from the EMS_SCADA tab)

Answers: 27% will do a components test, 24% will run simulations, 13% will do outside testing, 28% will do vendor verifications and 8% have opted for the ubiquitous OTHER.

6a. What is the status of your Y2K operating contingency preparedness?

Will it include power system Y2K studies including Scenario analysis, capacity shortages/overages?

Answers: 65% have a plan, 14% have a plan and have actually drilled on the plan, and 20% have started looking and 2% are completely clueless.

Are you looking at Y2K blackstart/restoration plans?

Answers: 58% have a plan, 36% have tested and drilled the plan, 9% are looking into it and 1% haven't a clue.

Now for another Gem from the General Planning tab:

11a. Are you making your NERC report available to the public? Answers: 35% Yes, 65% No

11b. Are you making your NERC report available to state regulators? Answers: 71% Yes, 29% No.

Damn.

So the conclusions that I draw here are thus. Everyone is running behind. At this late date, much work is still to be done and yet we're telling the general public happy happy happy news. In the mean time, we have been bullied by the state into telling the truth and they are "in the loop". This fits with the emergency planning session I heard about within my own state in which the state EMA officials were quite serious about preps, Nat Guard, panic scenarios etc. This was back in March. They know the truth, and you dear reader don't.

How does it feel to be part of the Sheeple.

-- Anonymous, July 31, 1999

Lane, your statement about a reasonable scenario not always equating with true put me in mind of a short segment I recently saw on the History Channel as part of the "Century". Several people of note had been asked for their thoughts about what will happen in the next century. I came into this piece rather by accident and I truly wish I could remember the man who made the comments which I found so logical. He mentioned that while we can comprehend the past, we can't affect or change it; and while we can affect or change the future, we can't comprehend it. His mathematical-based reasoning was that while predictions can be made about various future aspects, there is a _much_ larger numerical base of interactive imponderables which will always engender surprises. Thus, his prediction for the next century was that we will all experience profound surprise. While this was not a commentary on Y2K directly, it seemed to me to be a very good summation of the issue nevertheless - and corresponds very well with your thoughts.

-- Anonymous, July 31, 1999

Tim,

I'm quite suprised by the use of GPS in USA generation and transmission facilities for time keeping (particularly for SCADA and synchronising). This is something that is completely new to us here in NZ, so I do have a few questions.

Is the practice widespread in USA? As in what percentage of generators are using GPS rather than ground based timekeeping systems.

What advantage is there in using GPS?

And why go to all of the extra trouble of matching satellite systems to SCADA anyway?

Thanks

Malcolm

-- Anonymous, July 31, 1999

Malcom, you'll likely get responses from industry people, but I can offer the link to the TrueTime Power Utility page on GPS receivers, and their short description of what GPS clocks can do for utilities:

"TrueTime specially configured GPS and GOES synchronized clocks are designed meet the particular needs of power utilities. These include frequency and time measurements as well as precise time and frequency outputs to SERs, fault locators, and phase angle systems."

http://www.truetime.com/DOCS/TTprod_litFRM.html

For you, or any other readers, there is also a comprehensive site about GPS, with a lot of explanations and links about the entire end-of-week rollover problem, what it can effect (telecoms, banking, etc.), the uses of GPS - Navigation, Mapping, Tracking, Location and Timing, and also a link to a Canadian "Average Frequency Traceability Techniques" and much more. It's at:

http://www.sustainableworld.com/y2kgps/gpseng/index.html

The link there which leads to:

http://www.trimble.com/gps/puttinggps/gpsfram1.htm

will bring you to another link in the Timing section, "And a major Pacific Northwest utility company makes sure their power is distributed at just the right time along their 14,797 miles of transmission lines."

Click on that link and there's even a tutorial with info about "High Energy Transmission with High Precision GPS Time" with information about the use of GPS by electric utilities from the Bonneville Power Administration in the Pacific West of the United States.

The following is part of what Bonneville wrote about GPS:

"Given the amount of power a power company supplies, the number of customers they service, and the high-precision timing that the generation and transmission of electricity requires, it's surprising that power doesn't fail more often. Increasingly, GPS is becoming the tool that is used to provide that timing."

"Since 1988, the Bonneville Power Administration in the U.S. Pacific Northwest, has been integrating GPS technology into its operations. As an integral part of any electrical operations system, timing is the technology on which many of its functions are based. Generation and power transfers are planned in advance. Utilities coordinate with each other by making adjustments on a GPS timed schedule. Outages for maintenance are scheduled to ensure that they do not interrupt reliable power delivery. Disturbance records are aligned with recorded GPS time tags for analysis and comparison with related information. Price varies with demand so even billing is based on time. Advanced applications like locating power line faults (short circuits) and real-time phase measurement require continuous timing with high precision. And bad timing can throw a monkey wrench into all these operations."

"BPS Administrator, Kenneth Martin, puts it all into perspective. 'With help from GPS we are finding ways to develop a comprehensive system that meets the needs of new applications while continuing to serve existing systems. In short, we have found that GPS is the universal answer for power system timing, meeting all requirement of accuracy, reliability, coverage, and cost.'"

Unfortunately, the "universal answer for power system timing" had some unforeseen problems which needed to be addressed.

-- Anonymous, July 31, 1999

Bonnie,

You are absolutely fantastic. I don't know how you are able to find so much quality information so quickly, but I am sure thankfull for what you are able to do. If you ever decide to come to New Zealand and work as a researcher you would be most welcome.

Regards

Malcolm

-- Anonymous, July 31, 1999

I love it when a plan comes together with facts, but do I remember it was Rick Cowles that said GPS rollover would not have an effect on the power system?

-- Anonymous, July 31, 1999

On many occasions during interviews (and when questioned privately), I've said that I think the GPS issue is pretty much a non-starter for the electric industry. I still believe that. I've synched generators before - no timing involved - a simple waiting for 12 o'clock on the synch meter, and throw the switch.

(For those of you not familiar with this operation, picture this: you've got a generator spinning full tilt boogie and ready to load. You're the operator. You're watching the synch meter spinning around - basically, it looks like a clock, with only a minute hand, and you're watching that minute hand spinning around. Your mission is to throw the synch switch when that minute hand reaches 12 o'clock on the meter; maybe a few "minutes" before or after. Not to hard, eh? When you do that, your generator / transmission line / distribution substation will be "in synch" with the grid.)

I really don't want to get into electrical theory, 60 cycles per second, etc. etc. Clock time has nothing to do with electrical system synchronicity that I am aware of.

I have not been shown a single reason why GPS would have an impact on the ability to conduct synchronization of any element of power delivery.

-- Anonymous, July 31, 1999

Bonnie (and Tim),

I have now had a chance to follow the links on the GPS based clock, but I cannot see anything there that is critical to any form of power generation and transmission. As I said earlier, we do not use GPS for any form of timing here in NZ, and from what I have seen from your links I do not believe that would be any advantage in using such a system.

We carry out our time checks using a master/slave clock setup, whereby the master clock is sychronised to the national time clock at the institute for nuclear and geological physics) automatically via time pips that are broadcast hourly on a national radio frequency. A primary backup to the radio signal is a timed pulse which can be sent via PLCs, and a final backup is a phone call from the national dispatcher saying "the time error is now 0.xx seconds fast/slow. The slave clock compares frequency time to actual time and a time error is then displayed. It is a very simple system which does not require any form of high technology, and is not very expesive to install.

From what I can see the GPS clock simply performs the same function as our master clock. The consequences of the GPS system failing are minimal. The frequency can still be maintained at 60 Hz, time error (which is much tighter in USA than NZ) can still be kept to within acceptable levels, and the GPS clock is not used for plant synchronising anyway.

So overall, there will be no real effects on the grid should GPS fail.

But I still appreciate the work that you have put in to bring this matter to everyone's attention.

Malcolm

-- Anonymous, August 01, 1999

Factfinder, you've got a phaser? Is it a Federation model or a Romulan one? *grin*

Before you go into warp drive, notice that my post was addressed to, and in answer to, Malcolm's questions about the use of GPS in U.S. utilities and the potential advantages of it. I didn't even mention grids or synchronization! I also thought you'd be aware that I've stated before in this forum my agreement with Jim Lord's opinion that, "Most at risk here are small commercial fishermen, trucking firms, independent shipping companies and all recreational users. Foreign flagged ships visiting U.S. ports would likely be another concern."

I also posted this answer to someone in January of this year:

"Mike, according to Rick Cowles in his interview onsite at:

http://www.cbn.org/y2k

most utilities are now aware of the GPS problem and are dealing with it. He also mentioned there are other ways to do time synchronization of the grid. From what I've read recently the GPS problem is expected to be greatest in small municipalities who might not be aware of the need to adjust for the rollover in August, smaller shipping lines, overseas areas where awareness is lower, and on the individual level. It's not a complicated problem to fix, it just takes awareness of the need to do so."

Since that time, I'd add some smaller surveying companies and emergency service tracking in some communities to the list of where receiver upgrades might not have been made. Now, you don't happen to have a holodeck hidden somewhere, to go along with that phaser do you? You could program a "Gridzilla" monster and do battle! Slash! Parry. Punch! Take that, you foul dragon! Hey, if it could help a Klingon work off frustration, there may be something to it...*wink*.. and afterwards you could keep the phaser on stun and not have to vaporize anybody. Then again, disassembling someone's molecules does have its advantages...(it's not messy) Ok, Ok, I'll quit before the Star Trek lure completely overcomes me...

-- Anonymous, August 01, 1999

His mathematical-based reasoning was that while predictions can be made about various future aspects, there is a _much_ larger numerical base of interactive imponderables which will always engender surprises. Thus, his prediction for the next century was that we will all experience profound surprise.

Thanks for the input, Bonnie. This reminds me of what Drew wrote in one of his commentaries: that Y2K is a "statistical infinity" which renders accurate predictions impossible.

"Profound surprise". I have been thinking lately that we are probably going to have some really big surprises. Some pleasant, some not. That is, some things that we are worrying over obsessively are simply not going to be a problem, but some things we haven't even thought about will turn out to be nightmares.

I think that this whole GPS thing is just one more way that Y2K Alarmists are shooting themselves in the foot. (See "Trigger Dates" in Y2K Red Herrings.)

Nice HTML, FF. :-)

-- Anonymous, August 01, 1999

It seems the main point of my article is being missed. No where do I say that the GPS alone will cause a failure. What I am saying is that this event will begin a proccess. I tend to agree with the position that by itself, the GPS would have little effect on the grid, however, it does seem to me that the loading of straws upon the camels back will begin then. There are many other factors to consider, and other systems likely to experience disruptions, and any one by itself is certainly managable. It is the cummulative effect of many small errors that concern me, and clearly concern many of our leaders, up to and including the president. I say over and over, no one really knows, or can know what the future holds - but we can take an objective look at the many factors that may conspire to cause a serious problem, and with this type of rational, fact based information, make prudent preperations. I have no intent to take, or defend any particular position. Objectivity is my goal. My article presents some facts that have been completley ignored in this thread so far, the GPS is only one small part of a much bigger picture. I thank all of you for the thoughtful, insightful comments so far, and would truly appreciate more commentary about the other issues raised. Thanks again, Tim

-- Anonymous, August 01, 1999

August 1, Year of our Lord nineteen hundred and ninety-nine.

Most of the date instances in the NERC spreadsheets are two digit, both date variables and literal strings ("3Q99"). Uh, I don't think there's a windowing algorithm going on here. will be inyteresting to see when NERC claims internal compliance, 4Q99 perhaps. What a hoot.

-- Anonymous, August 01, 1999