NEED HELP ON OLD NEWS ITEM--EMBEDDED CHIP

About a 1 1/2 years ago, I remember reading on a website (c.s.-year2000? old Gary North site?) about a heavy industry factory that did a y2k remediation, advanced the clock to 1/1/2000, and re-started operations. Shortly after the operations automatically closed down. The cause wasn't found until several days later, when they discovered an embedded chip in the top of the factory's main smoke stack: the chip had detected the smoke and told the system to shut down, because the stack was now overdue for its routine maintenance (cleaning.)

Does anyone with a better memory than mine remember when and where this happened, and whether I have the details correct? Tnx in advance.

Bill

-- William J. Schenker, MD (wjs@linkfast.net), May 03, 1999

Answers

This may not be the article you are looking for but it is a good one
 

 Fortune: 4.27.98 Industry Wakes Up to the Year 2000 Menace

http://www.pathfinder.com/fortune/1998/980427/imt.html

-- Brian (imager@ampsc.com), May 03, 1999.

Bill:

Sounds like a legend or a hoax, especially if it was on the North site. What you describe sounds like a CEMS application, common in most smokestack industries. Here is an excerpt from the latest NERC report about CEMS in power plants:

Emissions Monitoring - There are 18 organizations reporting 33 units that are pending upgrades and certification of Continuous Emissions Monitoring Systems (CEMS) after June 30. The reporting entities are heavily dependent on vendors of these systems for upgrades and certification. There appears to be a backlog of demand for vendor support in this area. These systems provide a monitoring function only and do not affect the ability of any unit to produce electricity. These systems do not effect the ability of a unit to operate and they do not effect the quantity of emissions from a unit. The impact is principally one of potential regulatory penalties if an organization is forced under emergency conditions to operate without CEMS.

As you can see, these are monitoring functions only and would not trigger a shutdown if they were non-compliant or simply failed to operate at all (as happens routinely now).

-- RMS (rms_200@hotmail.com), May 03, 1999.

Hi Bill,

The test shut down a power plant. I'm sure there will be an article on Rick Cowles message board about it.

-- Dean -- from (almost) Duh Moines (dtmiller@nevia.net), May 03, 1999.

William:

Out of sheer curiosity, what use do you intend to make of this old information, assuming you find it?

-- Flint (flintc@mindspring.com), May 03, 1999.

In keeping with the spirit of embedded stories, of the potential kind ...

Checked out some manafacturing, process control insurance company web- sites.

Factory Mutual has some interesting Y2K glitch-like stories to illustrate potential problems ...

... there is the potential for serious equipment and facility damage as a result of minor glitches in computerized equipment, shutdowns of systems, and/or loss of utilities. ...

http:// www.factorymutual.com/y2kintro.htm#bug

The following examples, while not necessarily related to Year 2000 date problems, illustrate the types of potential exposures a business could face as the result of Year 2000 induced process control problems. This is not intended to infer that these are the only Year 2000 process control issues possible. -- Factory Mutual

http:// www.factorymutual.com/y2kdamge.htm

Explosions

A malfunctioning programmable controller operating a fuel mixer dropped the mixing bowl and turned on the mixer. This caused the mixer blades to contact the mixer bowl. Friction ignited the mixture in the mixer bowl. Ignition caused an explosion damaging several buildings.

###

During an on-line change of a circuit board in the boiler burner management computer at this coal-fired electric generating station, a faulty board was installed which caused the inlet dampers on both induced draft fans and one of the two forced draft fans to close to a minimum. The unaffected forced draft fan pressurized the boiler and initiated a master fuel trip. Hot gases backed up the coal transport pipes to the coal pulverizers where coal dust was thrown into suspension and was ignited by the hot gases. Explosions occurred in the pulverizers. The installation of the faulty converter card in the boiler burner management computer allowed the computer to receive indication that the ID fan dampers were closed.

###

A software-controlled PLC solvent coating operation had a computer card that did not function properly. As a result, methyl ethyl ketone discharged into an idle process tower. This was unknown to the operators. This flammable liquid accumulation allowed vapor in the process equipment and duct to exceed the lower explosive limit.

###

Fire

A malfunctioning control circuit caused a fire at an unattended test assembly. The test assembly started on an automatically controlled 24- hour test with sampling every 4 hours. Later, fire was noted in the building. Most probable cause of the fire is a malfunctioning programmable controller that allowed an immersion heater to heat the oil to its flash point.

###

Loss of control

A newsprint mill refiner and its driving motor were damaged due to voltage and frequency fluctuations. Problems began when several voltage and frequency fluctuations occurred in the mill's power systems. A control computer shut down causing operators to lose control of electrical machinery. Simultaneously, loud noises with heavy vibration and shaking could be heard and seen coming from the refiner. Parts were falling off the machine; the building was shaking. After 10 minutes, control was restored by resetting the control computer.

###

Misinterpreted condition

A facility was testing a boiler feedwater control loop for date rollover to Year 2000. The control console date was set in a fashion similar to testing a PC. It was changed to 12/31/99, 23:58, and then powered down. A few minutes later, it was powered back up. The only problem noted was that the year showed as 1980. The programmable logic controller loop continued to function normally. Boiler levels were simulated up and down to drive feedwater-regulating valves with no problems. Technicians reset the console clock to 12/31/99, 23:58, and did not power down. When the clock rolled over to 01/01/2000, there was no problem. They then powered down the console and restarted it. It rebooted with a date of 01/04/80. A downstream programmable logic controller apparently saw this as a major mismatch with its own clock and misinterpreted the condition. Feedwater regulating valves were driven shut, and the boiler trip logic initiated. In a live situation the plant would have tripped off line.

###

Overheating

An over-temperature protection system did not operate causing the partial melt down of three heat exchangers in a gas fired fume incinerator. There were several over-temperature circuits that should have operated automatically to trip the gas safety shutoff valves.

###

During normal operation, a furnace at an aluminum rolling mill was on semi-continuous operation (by computer). An error in computer software and the PLC caused overheating of loaded ingots that caused ingot bleed and damage to the furnace. An erroneous resetting of the counter reading associated with the PLC caused overheating. Prior to the incident the unit had been taken off line for a program change. When reactivated and placed back on line the counter was reset to "1," while the PLC counter remained at "30." The controller updated its tracking indexing by "30," and in effect, erroneously simulated removing all alloy ingots and inserting the next furnace load, which involved a different alloy. It then altered all the zone temperatures; however the initial alloys were still in the furnace since the PLC had a count of "30." No one realized the unit was reset to "1," while the PLC was on "30" count.

###

Process disturbance

Twenty-seven trays in the Argon section of a low-pressure distillation column were upset from their supports at an air separation plant. A process control computer malfunctioned and caused a process disturbance.

###

Shut down

Five pot cells were damaged by change in temperature to the equipment when a system's process control units shut down. Personnel could not restart the equipment, did not determine the cause of the shut down quickly to prevent freeze-up, and the melted aluminum solidified. Investigation revealed that 1996 was a leap year. Programmers had utilized a 365-day year.

###

See also ...

Examples of how Y2K could affect various industries

http:// www.arkwright.com/risk/y2k/effects.htm

The Year 2000 Issue
How Y2K Could Affect Your Industry

The following are examples of how the Year 2000 issue can affect your industry. They are scenarios to describe potential incidents that could occur and serve only to illustrate the complexity and magnitude of this problem, and do not have anything to do with insurance coverage ...

Electric Power Generation Station

A pulverized coal fired steam generating unit is operating under full load conditions when an erroneous signal is received by the control logic as the result of a Y2k date related mismatch. The logic in the control system initiates a master trip sequence. Safe and uneventful shut down of the unit is dependent upon all mechanical and electrical shutdown devices functioning properly while operating in the prescribed manner and sequence. Failure of these safety devices could lead to considerable property damages. The combustion air passes of the steam generators experienced negative pressure excursions due to improper cycling of the dampers and fans resulting in an implosion of the ductwork. The gas passes experienced an over-pressurization due to an accumulation of unburned hydrocarbons being ignited by the hot surfaces resulting buckled walls and buckstays. Delayed closure of boiler feed pump steam stop valves resulted in an over speed of the turbine.

###

Semiconductor Fabrication Facility

A 50,000 sq. ft. Class 1 clean room was shut down for the Year 2000 weekend because of managements concerns over potential production exposures from nonY2k compliant operating systems. The facilities security system was tagged out as inoperable due to some recently discovered date related problems. Thieves take advantage of the shut down period and inoperable alarm system to take in excess of $1million of finished semiconductor chips.

###

Integrated Steel Mill

A fully integrated steel mill is in full production through the Year 2000 weekend. The entire hot roll mill from the soaking pits to the slab run out table is controlled from the operator pulpit using a programmable logic control system. The controller stops functioning when it receives an erroneous signal as the result of a Y2k date related mismatch. The entire rolling mill operation comes to a complete stand still. An ingot stops between the first and second stand of the hot mill. The radiant heat from the ingot ignites grease and rolling oils. Fire ensues on the mill damaging electrical cables, local controllers, roll positioning motors and hydraulic oil lines.

###

Chemical Plant

A complex chemical facility is in full production. Process reactors are monitored, controlled and operated using distributed control systems. Process flows, reactor residence times, temperatures and pressures are monitored and controlled by this system. Timed addition of reactor ingredients and catalysts are critical to controlling some of the process reactors. A distributed control system receives an erroneous signal as the result of a Y2K date-related mismatch. A catalyst is added too soon to the reactor cycle. This results in a process upset leading to an uncontrolled exothermic reaction resulting in an explosion damaging reactors and its associated equipment.

###

Interesting examples, huh?

*Sigh*

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), May 03, 1999.

Diane:

These are very interesting. Some are speculative, some are operator error, some are non-date computer bugs, some are mechanical breakdown. It's an imperfect world out there, for sure.

Let's hope most of the y2k embedded testing is done both safely and effectively. Reports of problems found during this process are somewhat reassuring first because problems found are presumably fixed, and second because a lot of this equipment is standard, and problem reports can be sent to all users of similar equipment, speeding up the remediation process.

-- Flint (flintc@mindspring.com), May 03, 1999.

ANSWER TO FLINT'S FIRST POST:

If it turns out that the incident actually happened as I remember it, it can act as a guidepost in determining how big a problem embeddeds may turn out to be, if all are not uncovered prior to y2k. There appears to be a paucity of publically available info regarding ACTUAL EXAMPLES OF EMBEDDED FAILURES CAUSING REAL WORLD PROBLEMS. Aside from the recent article in WIRED, describing Texaco's chip remediation program, there appears to be a fact vacuum. Within this vacuum I hear a lot of noise and static from both the Gloomers and the Pollys -- I digest facts better than rumors, no matter how they fall.

Bill

-- William J. Schenker, MD (wjs@linkfast.net), May 03, 1999.

Bill:

It's important to use such an example (and *some* such examples have been real, even if the one you recall was speculative) to illustrate two things: That such problems exist, and that they are being fixed when discovered.

To put it more directly -- tales of isolated failures turned up by tests a year or two ago are a *long* way from a description of where things stand today. So I was a bit concerned that you might be looking for obsolete information as part of a (perhaps) well- intentioned disinformation campaign. If I'm wrong, I apologize.

It might be helpful for you to contrast the article about GM in Fortune Magazine last year (lines stopped, punch clocks failed, security system died) with the latest reports of GM plants now passing all tests. A lot has happened in a year.

-- Flint (flintc@mindspring.com), May 03, 1999.

I recall the story, but not the source. It was reported to have occured in the UK at a power plant. It may not have been an "embedded chip", but it was a process monitor system that attempts to calculate amounts of particulate matter going up the smoke stack. The report stated that it did trigger a shutdown of the plant, but I do not recall the length of time before restart.

Jerry

-- Jerry B (skeptic76@erols.com), May 03, 1999.

ANSWER TO FLINT'S THIRD POST

Yes, I can see where you're coming from, Flint: the level of acrimony from both sides of the isle increases the 'paranoia index' of us all.

No, I don't have a 'campaign' in mind, altho I'm a certified G&D'er from way back -- even though a fact may jar my G&D 'religious faith' (and any strong conviction has that characteristic), I still can sleep better at night if I face up to facts. In the last few weeks my G&D rating has dropped from a 10+ to maybe an 8. It may drop some more. I am looking at the same graph you are: I want to see more than 1 point on a curve --- a snapshot of any situation doesn't give anything like the info that a time sequence does.

Flint, we are on different sides of the isle, but the distance between us may be shortening. Keep up your desire for the Truth, & I'll try to do the same.

Bill

-- William J. Schenker, MD (wjs@linkfast.net), May 04, 1999.