An earlier thread started by Drew Parkhill mentioned the article on year200.com. I thought it might be interesting to talk about the DCS issue that that article spoke about.

The portion of the article by Daniel Dolan that I found most interesting I have copied below, and below that I have copied a few paragraphs from the new NERC report. Dan the Power Man mentioned that this new NERC report should/would? "answer" much of the questions raised in Daniel Dolan's article. I'm not sure if this issue is one that was being implicitly referred to, but if so, I am not comforted by the "answer" to the questions raised about DCS. Don't get me wrong though, I understand the problem that the power companies have in getting reliable data from the many telecommunications companies on whom they rely. This really is what constitutes my most significant concerns about power (and for that matter all of the basic infrastructure areas -- how can we know if we have received reliable data from all of the interconnected vendors?). Yesterday on my local radio station, I heard a quick news-byte that basically said something like "NERC says the electric utilities in this country are 99 percent complete in doing the work to assure us that there will be no Y2K caused problems with electricity". Sorry I can't remember more exactly the specific words used. The thing that bugs me is that the radio news didn't say anything about "providing of course that the telecommunications vendors get all their issues taken care of, and the work outlined in the exception categories turns out to be actually done with no additional problems turning up there".

This post is rather long, just to try to keep your attention here though is the statement from the new NERC report that seems to hit me the hardest:

on page 23 (32 of 92 using Acrobat's page numbering)

It is apparent that extensive integrated testing with external voice and data communications service providers is not practical. Typically, these service providers are working hard to complete their own program and cannot dedicate substantial resources to joint testing with individual customers, including electric utilities. Also, these service providers typically cannot provide live circuits for end-to-end testing with electric systems, leaving most testing for the laboratory.

*------* Below is excerpted from the article written by:

Daniel P. Dolan Assistant Professor and Senior Research Fellow Center for Global Communications

on www.year2000.com: http://www.year2000.com/y2kcurrent2.html

...

Far more at risk of Y2K failure are telecommunication services and equipment and electricity production facilities, especially those facilities with Digital Control Systems (DCSs). Telecommunications systems are used by utilities to monitor real-time performance of electricity transmission and distribution. Data communications are critical in providing generating units and control centers with the information required every few seconds to balance electrical generation to demand. However, according to NERC a "significant portion" of voice and data communications used by electric utilities is provided by telecommunications companies, many of which lack the resources to conduct extensive tests of their equipment and services before 2000. These include local and long distance carriers, Internet service providers, paging and cellular systems, and satellite broadcasters. The result is that utilities have little control of the Y2K readiness of their data communication systems, putting in question the ability of electric utilities to manage the real-time functions of the power grid without interruption. NERC therefore advises electric utilities to install Y2K-compliant telecommunications systems if possible, and to be ready for manual operations of systems if necessary. DCSs control nearly all aspects of power generation, automating what are otherwise very labor-intensive processes and procedures. DCSs contain many embedded computer chips and, according to NERC, rely on time-dependent algorithms that may result in unit trips. Perhaps most critically, DCSs are "vendor-dependent," meaning that electric utilities must rely on the expertise and personnel resources of DCS vendors for cooperation in Y2K testing. NERC has publicly admitted that, as with telecommunications providers, DCS vendor resources are not adequate for testing and upgrading of all critical DCSs before January 2000. Another reason that DCSs are of great concern is because they have the potential for causing what power engineers call "common mode failure." Common mode failure in electrical service is most often associated with weather, for example the 1977 New York city blackout and the Auckland, New Zealand blackout of 1998. However, DCSs could become sources of common mode failure in electricity production facilities if critical and Y2K-sensitive embedded chips are not discovered and replaced in time. ...

*-----* Now below here I have copied some paragraphs from the new NERC report found when I used the Find function of Adobe Acrobat to locate "DCS" :

starting on page "-v-" (6 of 92 as reported by Acrobat reader)

Key Issue 4: Supply Chain Dependencies _ Close analysis of the exception items in Appendix C highlights the electric industry's dependence on suppliers of critical technologies associated with power plant distributed control systems (DCS), energy management systems (EMS), supervisory control and data acquisition systems (SCADA), and communications hardware and software. Although the large majority of suppliers have been responsive to the needs of the electric industry, there remain some suppliers that are not fully cooperative or are stretched beyond their resources. The electric industry continues to seek upgrades, testing, and certification in some of these critical areas from particular vendors. NERC will continue to report on the status of vendor support along with future updates on the Exceptions List. If necessary, NERC will report to DOE the names of individual vendor organizations requiring attention.

on page 13 (22 of 92 from Acrobats numbering)

Despite these assurances, exceptions related to SCADA/EMS, plant DCS and controllers, and communications are essential to reliable operations and warrant close scrutiny in the follow-up tracking process by NERC and DOE. It is apparent also that the greatest uncertainty in the area of exceptions is the ability of vendors to meet these stated schedules. Recommendations: 1. Organizations reporting exceptions are requested to provide a monthly status report on each item in the exception report. NERC will publish a monthly update of the exceptions summary list. 2. These organizations should make a concerted effort to meet the stated schedules.

on page 23 (32 of 92) The electric industry owns and maintains a majority of its voice and data communications facilities. However, external service providers are used for a significant portion of voice and data communications. These external providers may be local telephone carriers leasing dedicated circuits to carry monitoring and control signals to power plants and substations. They also may provide long distance services, satellite systems, cellular systems, and wide-area networks. The electric industry, like many other industries, is dependent on a complex set of integrated communications systems.

Most entities report satisfactory progress in testing their internal communications systems, as reported above. Like EMS/SCADA and DCS systems, communications is an area that often requires support from vendors. Entities report making Year 2000 upgrades on older network equipment (e.g. routers, hubs, and switches). Often, testing procedures or results have been achieved with the assistance of or information available from equipment vendors.

It is apparent that extensive integrated testing with external voice and data communications service providers is not practical. Typically, these service providers are working hard to complete their own program and cannot dedicate substantial resources to joint testing with individual customers, including electric utilities. Also, these service providers typically cannot provide live circuits for end-to-end testing with electric systems, leaving most testing for the laboratory.

Examples of areas where Y2k anomalies have been discovered in electric utility owned communications systems include:

7 Network management software 7 Routers - primary functions work, diagnostics software may be impacted 7 Control Signaling Unit/Digital Signaling Unit devices - incorrect date display 7 PBXs - some require remediation 7 Fax machines - incorrect date stamp

Partial loss of voice and data communications remains a high priority for contingency planning for electrical systems. Backup voice communications systems that do not have common failure modes with primary systems are the appropriate strategy for voice communications. These issues are discussed further under contingency planning in Section 4.

Recommendations:

1. Any remaining electric utility owned communications systems that have not completed Y2k testing should be completed as soon as practical and the final status reported to NERC.

2. Controls should be adopted that assure Y2k Ready facilities remain ready.

in the Appendix C of the NERC report entitled : "Summary of Non-nuclear Exceptions Reports" (71 of 92)

DCS is listed many, many times under the heading: "Facilities, Components, or Devices" where the specifics for the exceptions are listed.

-- Anonymous, August 04, 1999

Answers

Dennis, the only contribution I can make right now is to give you a quote from an EPRI representative's statements at a NERC Y2K Workshop held in October of last year. It's applicable to your concerns and certainly food for thought.

"He indicated that testing of distributed control systems can give ambiguous results; systems tests are very difficult to perform in power plants; and system tests are almost impossible in the Transmission and Distribution areas."

ftp://www.nerc.com/pub/sys/all_updl/docs/y2k/workshop-minutes.pdf

-- Anonymous, August 04, 1999

dennis, to me the following translates into -- not practical... translate this into no can do cause they are *not* ready; cannot dedicate substantial resources... translate this into no can do they are *not* ready[this, despite, i am sure, all the political pressure that was brought to bear in order to assure the stability of the electrical grid... also would *not* look good if they fell on their collective faces in a public arena.]

the laboratory testing is confusing to me... if 'they' were ready why not just go live?

dennis, there was a report from the commerce department on the fragility of vital services in the international community... this was about 3 or 4 months ago, and i can't find it in the archives.

i had done a post on the telecommunications section of the report and one of the things that i noted was the careless conclusions made by the department when commenting on the international community. in that particular report it was noted that the more technologically advanced nations had the most to fear regarding the telecommunication's industry. it also mentioned that demographic intensity made the problem that much more complex and difficult to remediate.

guess who is one of the most technologically advanced and dependent nations in the world? now quess who, of the larger nations, has the largest number of phones per capita?

did you buy your ki yet?

It is apparent that extensive integrated testing with external voice and data communications service providers is not practical. Typically, these service providers are working hard to complete their own program and cannot dedicate substantial resources to joint testing with individual customers, including electric utilities.

Also, these service providers typically cannot provide live circuits for end-to-end testing with electric systems, leaving most testing for the laboratory.

-- Anonymous, August 06, 1999

...and another thing jordan.

the one thing that does keep me going is the inordinate amount of email that i receive from 'the silent majority.'

there are many out there who are loathe to post in a public forum... but it does not mean they do not think and it certainly does not mean that they do not have opinions.

there are less now that rick imposed the password... but they are still out there and they will all remember.

god bless the politicians and the bureaucrats if the worse comes to pass.

-- Anonymous, August 08, 1999