Y2K bugs hidden in chips: Experts don't expect a technology disaster

By Clint Swett Bee Staff Writer (Published April 7, 1999)

Armed with flashlights and screwdrivers, Ron Neese and Kevin Brown are stalking electronic gremlins.

They pry the covers off control panels, peer into storage closets and inspect elevator equipment rooms inside the basement of the California Highway Patrol headquarters on First Avenue.

Their exhaustive search is for tiny computer chips that could cause havoc on Jan. 1, 2000, the date when the so-called Y2K bug -- which renders some computers incapable of correctly identifying any date after Dec. 31, 1999 -- is expected to strike.

But the bug is not only lurking in PC software or on giant mainframe computers. It's hiding in millions of embedded processors, tiny computer chips that take their name because they are "embedded" into larger systems.

"We've found some problems in every building we've inspected," said Neese, service manager for Mark III Engineering Contractors, a Sacramento engineering and construction firm conducting embedded chip inspections for several state agencies.

Billions of embedded systems exist around the globe, running everything from navigation systems on passenger jets, to VCRs, automobile engines, building security systems and slot machines.

All these chips get their intelligence from the software instructions that are burned into them during the manufacturing process, and therein lies one of the biggest problems.

Y2K sleuths can't look directly at the programming code on the chip as they can on a computer. They have to rely on the system manufacturer to tell them if the chip is vulnerable to Y2K glitches. They also can search for glitches themselves by turning the clock on the systems forward to 2000 and wait for a reaction.

The good news, experts say, is that the results aren't likely to be catastrophic.

"The embedded system problem is turning out to be much smaller than we had feared," said Peter de Jager, a computer consultant and one of the first to sound the alarm over the Y2K issue.

Of the billions of chips worldwide that control various devices, the majority "don't even know what planet they are on, much less care what date or time means," said Andy Kyte, who tracks Y2K issues for the Gartner Group, a technology research firm.

And even most of those chips that are programmed to be date-sensitive won't be affected by the turn of the century.

For instance, controllers used in lawn sprinkler systems don't care what year it is, only the day of the week.

Neese, of Mark III, said the most common vulnerabilities he's seen in inspecting more than 150 commercial buildings are card-key security systems that might not let people into the building, heating systems that might not fire up and voice mail systems that could lock up a firm's phone network.

"It's hard to say how critical those systems are," said Neese, "but people who don't have phones or heating probably won't come to work."

Still, the Gartner Group estimates that the vast majority of vulnerable silicon devices are in industrial processes, such as factories and utilities.

"These tend to belong to large corporations, which have the resources to identify and resolve the problems before they occur," Kyte said.

For example, some date-sensitive chips may control welding robots on a factory floor. A robotic shutdown would be inconvenient for the company, but hardly a disaster for the population at large.

Chips that control critical functions involving public safety, such as traffic signals, may be vulnerable. But municipalities already are making sure their systems function at the turn of the year.

The city of Sacramento is conducting an audit of all its embedded systems to see which ones could cause problems Jan. 1, 2000, said Sally Nagy, the city's chief information officer.

So far, technicians have discovered a couple of areas of concern. They replaced some chips in the city's switchboard system, and said some devices that control door security systems in city office buildings may need attention.

The city also is investigating whether embedded chips in police car terminals, which are used to check car license plate numbers and the like, may have the computer glitch, Nagy said.

SMUD is doing a major review of its embedded systems to keep electricity humming through its wires. "We have a very simple approach," said Winston Shizawa, SMUD's chief information officer. "If it has a wire coming out of it or a battery attached to it, we check it."

The biggest problem the utility has found so far is a chip that controlled an emissions monitor at one of its power plants, Ashizawa said. When tested to see if it recognized the year 2000, the chip sent a false signal telling operators the emissions monitor wasn't operating correctly.

"Then you are faced with decision of running the plant and ignoring the warning or shutting the plant down," Ashizawa said.

He said SMUD, which serves nearly 500,000 businesses and homes in Sacramento County, is working with the device's manufacturer to correct the problem and he said all fixes and testing on critical systems should be completed by September.

"We know that even minor flaws will shake the public's confidence, and we don't want that to occur," Ashizawa said.

One of the biggest fears of embedded chip failure is in the medical field, where microchips control machines such as heart monitors and devices that feed fluid intravenously to patients.

Sutter Health surveyed thousands of devices at its hospitals and found a few that will not operate properly come 2000. Those are being modified or replaced.

De Jager cited a Veterans Administration hospital that found only one device out of 1,600 that behaved erratically because of Y2K issues. Another 23 merely misstated the date, but otherwise operated correctly.

One commonly expressed fear is of elevators plunging down their shafts in the event of a Y2K glitch. Elevators are designed to stop operating if required maintainance is not performed on time, and in many elevators, chips count the time between inspections.

But experts say, even in the worst case, an elevator would just return to the lobby and sit idle with its doors open if the chips falsely indicated that repair schedules hadn't been met.

Another story is that offshore oil rigs have embedded devices at the bottom of the sea that technicians can't reach. Come next Jan. 1, the rigs might suffer blow-outs.

But Kendra Martin, a spokeswoman for the American Petroleum Institute, called the scenario "another urban myth." She said widespread testing showed all underwater devices can be controlled from the rig or from shore.

"We aren't finding any Y2K problems, had they not been repaired, that would be catastrophic," she said. "We're finding things that would be annoyances or inconveniences."

That's the kind of thing technicians at Sacramento International Airport are finding, as well. There, workers are busy taking inventory of everything from runway lights to baggage carousels to make sure everything's up to par, said Bruce Mosley, deputy director of the airport.

So far, the worst thing airport personnel have found is that some of the older jetways -- the mobile bridges that connect the plane to the terminal -- might not operate correctly, he said. "There's nothing on the ground that will be a disaster if it doesn't work. At worst, it will just be an inconvenience."

Greg Young agreed. Young is the tech support manager for Z-World, the Davis company that makes embedded chip systems for a variety of uses.

"The embedded problem will be, at worst, a few days of discomfort," Young predicted. "More problems are created by (doomsayers) than by the chips themselves."

-- Norm (nwo@hotmail.com), April 07, 1999

Answers

It's a well-known fact that less than 10% of all chips are date sensitive. It could be even less than 1%. Even with these small percentages, though, there are millions of date-sensitive chips out there.

National Guard embedded systems page:

http://www.ngb.dtic.mil/y2k/closer.htm

-- Kevin (mixesmusic@worldnet.att.net), April 07, 1999.

Funny that this utility only found one problem. Here's a story that was posted a couple of weeks ago on a test at a hydro plant. I think controls at a nuclear plant may have a few more embedded systems to worry about. <:)=

**As part of an experiment last year, technicians at the huge Xingo hydroelectric dam on Brazil's Sao Francisco River set the dates on the plant's main computer forward to Jan. 1, 2000.

**What happened next is still sending chills through Latin America.

**"When they put the date forward, the whole control board went haywire," remembers Marcos Ozorio, one of the members of Brazil's presidential Year 2000 commission. "Twelve thousand warning lights flashed all across the board, with all kinds of alarm information."

**Technicians quickly switched back the date, and are now ferreting out the plant's Y2K bugs. But "if you had been surprised by a situation like this, what you'd have had to do is shut down the plant until you found where the failures were," Ozorio said. "Automatically you'd be taking off the energy board 30 percent of northeast Brazil."

-- Sysman (y2kboard@yahoo.com), April 07, 1999.

"Armed with flashlights and screwdrivers, Ron Neese and Kevin Brown are stalking electronic gremlins."

LOL They're looking for a needle in haystack! So they look and look and look...and conclude...IT MUST NOT BE THERE!

Yes, we will see just how "disruptive" thousands and thousands of failed embedded systems are...and how long it takes to repair or workaround them in a world economy that has been shattered by y2k.

-- a (a@a.a), April 07, 1999.

"The city of Sacramento is conducting an audit of all its embedded systems....."

when didt they start? last week? isnt it a bit late for assesments?

-- hunchback (quasimodo@belltower.com), April 07, 1999.

Well, it's good to know the chips aren't down at CHiPS HQ . . .

The evidence suggests that the embedded system problem isn't likely to be as bad as originally feared, but this article strikes me as rather trite. I'd feel a lot better if we were getting recent results from a firm like TAVA that specializes in embedded systems work (and is doing such for GM and several other large American companies), instead of glib quotations from IT people like de Jager and Kyte (Gartner), who might be out of their field here. Last fall I did see some of TAVA's data on its work for GM, etc.; the results indicated up to a 15-20% failure rate in some complex embedded systems used in industrial applications (computerized/roboticized assembly lines, etc), a finding in line with earlier studies conducted in the UK for the Office of Health and Safety. Even Gartner, in an October report in which it said that only 1 in 100,000 "free-standing" (?!) embedded chips would fail, admitted that up to 35% of complex embedded systems might be at risk. (Such systems typically have 10 to 50 or more chips, each.) Frautschi has suggested failure rates up to 15% in such systems. A year ago, GM's CIO reported (in a "Fortune" interview) "catastrophic" problems with embedded systems in all of GM's 135+ mfg. plants worldwide; the reporter who did the interview later confirmed to Yardeni that the CIO, Ralph Szygenda, in fact used the word "catastrophic" four times during the course of the interview. Evidently Mr. Szygenda was just being fanciful and melodramatic that day.

Re oil rigs: last spring Texaco reported that it was finding a 12% failure rate among the 10,000 or so embedded chips aboard a typical offshore oil rig; other large oil companies (Shell, BP, etc.) were reporting 10% failure rates aboard their rigs. Gee, was that all just made up? Or were all those failures just "annoyances"? A major article in the April 1998 issue of the trade journal "World Oil" noted that there wouldn't be time or resources to adequately check 70% of embedded systems in U.S. oil refineries. Well, what the happy hey, it now appears it wasn't necessary anyway, right? And when Deputy Def. Sec. John Hamre and Sen. Thurmond agonized over that little fact of life in a Senate hearing last September, they were worrying for nothing, right?

Frautschi and others have found that even embedded chips that don't need to "know" the year may still have an internal calendar; there have been literally thousands of small chip manufacturers, and "generic" chips were made that could be accessed/used in various ways. Thus, a chip that is measuring only cycles or intervals or whatever might still have an internal calendar running. (Unfortunately, because of type testing and reliance on inaccurate vendor statements, at least some of these chips are being missed.) The real question then becomes whether that chip's internal calendar was ever calibrated with the Gregorian calendar; if not, then the chip will still fail, but at some unknown time after 1/1/2000. Failures spread out in time will obviously be much easier to deal with than failures occurring almost simultaneously. Nobody really knows for sure just how large the "spike" of failures will be next January, of course.

Again, the current evidence does suggest that failures of embedded systems ain't going to send us to hell in a handbasket. But just as there now appears to be a trend to classify more and more IT systems as "noncritical" (and hence "we can worry about them later"), there also appears to be a trend to proclaim embedded systems problems an "urban myth." Ah well. Tell it to GM.

-- Don Florence (dflorence@zianet.com), April 07, 1999.

And yes, Mark III is obviously an embedded systems outfit, but I'd feel better if they had results to report from work on major companies, utilities, etc., not from the basement of CHiPS HQ or other building systems. I don't expect many general bldg. systems to go haywire next January.

And yes, Kyte (of Gartner) acknowledged the problem in industrial and utility settings; again, see Gartner's report last fall. But Kyte certainly didn't acknowledge the magnitude of the problem at, say, GM. Saying that a large manufacturer or utility company has the resources to throw at a problem doesn't automatically guarantee that the problem will get solved on time, especially if there is undue reliance on type testing and vendor compliance statements.

-- Don Florence (dflorence@zianet.com), April 07, 1999.