A Y2K Mystery: How Many Mistakes Are Going Unnoticed? (From NY Times)

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

August 30, 1999

A Y2K Mystery: How Many Mistakes Are Going Unnoticed?

By BARNABY J. FEDER

A growing number of companies and government agencies are claiming that their vital systems are ready for any and all Year 2000 dates, but life seems as hectic as ever for many Year 2000 technical workers in that advance guard.

Managers continue testing on the assumption that at least minor flaws lie buried in some "ready" systems. Previously unknown Year 2000 defects keep popping up in commercial software. And normal activities like adding equipment or software carry the risk of inadvertently introducing bugs into systems that had been cleaned up and tested.

"Everyone is testing and retesting," said Steven Kutnick, the Year 2000 project leader at the Northern Trust Company in Chicago.

How many crucial mistakes are being caught by testing -- or missed -- has become a great Year 2000 mystery.

Consultants and companies selling Year 2000 software-testing tools contend that repairs typically miss many date problems and introduce bugs that may or may not be related to dates. Some authorities, like Capers Jones, chief scientist at Software Productivity Research, a consulting company in Burlington, Mass., warn that testing fails to catch many defects.

The Gartner Group, a technology consultant in Stamford, Conn., says it is common for independent tests to find errors lurking in 5 percent of computer code that has been altered during Year 2000 work and that the error rate reaches 15 percent in poorly run projects. Nearly two-thirds of those errors are serious enough to create faulty data or cause programs to crash, according to Gartner.

There have been a number of reports highlighting the consequences of failing to screen out defects. Northwest Metrology, an electronic test equipment company in Bakersfield, Calif., recently disclosed that an office computer it had tested last year had nevertheless crashed and destroyed data in January after encountering Year 2000 dates. The company, already out $80,000 as a result, is still struggling to restore the lost data eight months later.

Year 2000 project managers agree that such perils are real and that they make rigorous testing worthwhile. But many say their experience in fixing programs, testing them and putting them back into use leads them to doubt that hidden defects are anywhere near the levels projected by Jones and influential consultants like Gartner.

"We had to change less than 1 percent of our programs after testing," said Mary Stewart, who heads Year 2000 work at Total System Services Inc. of Columbus, Ga., which handles 195 million credit- and debit-card accounts for major banks and retailers like Sears, Roebuck and Nordstrom.

The Chase Manhattan Corp. and its outside consultants found just 300 errors after the bank made needed alterations to three million lines of computer code. "They tended to be extremely easy to fix," Brian Robbins, a Chase senior vice president, said.

Still, testing strikes Year 2000 managers whose repairs are completed as a good bet to minimize risks, if only by increasing confidence among business partners and consumers in general. Chase, in fact, went so far as to pay for independent testing of the repairs at the Alltel Corp., a key telecommunications supplier.

No testing program can cover everything software programs do, so managers like Ms. Stewart have allotted time to other forms of Year 2000 double-checking too, including assigning some programmers to what could be the most mind-numbing of all follow-up work, a process called desk checking. That is techno-speak for the job of reading through critical code line by line to search for possible errors that did not show up during the repair work or earlier rounds of testing.

In between the elaborate world of testing and the drudgery of desk checking lie a range of other tasks for Year 2000 technical workers finished with critical repairs. Some are repairing systems that had been on the back burner because businesses regard them as useful but not vital.

CSX Technology Inc., a unit of the Richmond-based CSX Corp. that has handled Year 2000 work for the CSX railway, its sister company, is about to repair some desktop programs it could do without, according to Charles Wodehouse, president of CSX Technology. Such "dogs and cats," as Wodehouse calls them because they are like old pets, produce reports or do other tasks that could be done on newer software like Microsoft Office. But it can be a battle to persuade managers to let the old programs die, he said.

At some big companies, retiring programs has become a time-consuming late-stage Year 2000 activity in its own right. Household International, a consumer credit company in Prospect Heights, Ill., is halfway through a two-month drive to shelve more than 10,000 programs it replaced with new software rather than fixing, according to Thomas Wilkie, head of the company's Year 2000 program. But the shelved programs will be archived for a year before they are discarded.

"You don't want to eliminate anything that might be used," Wilkie said.

Wilkie and others are also busy repatching systems that they had classified as ready as major software vendors like Microsoft report previously unknown Year 2000 defects. Microsoft recently released supplementary code, known as a patch, that represented its sixth attempt to make its NT operating system fully compliant with Year 2000 needs.

If a patch released by a vendor is deemed to be crucial, a company might have to undergo a new round of testing and training. Microsoft has been criticized because scarcely a month goes by without announcements of newly discovered flaws in popular programs like Windows, the NT operating system or the Excel spreadsheet, but it has plenty of company. Software vendors have posted nearly 480 warnings of previously unreported Year 2000 flaws since early May, according to the Infoliant Corp., a Pittsburgh firm that tracks such data.

A related concern -- benignly called "clean management" -- is preventing the software and systems the company has worked so hard to fix from being infected with Year 2000 problems by new, untested software or links to the outside.

The dangers are substantial for companies like Public Service Enterprise Group, parent of New Jersey's largest power company. Public Service finished Year 2000 work on its giant customer information system in January. This year, though, Year 2000 experts hovered anxiously as it rewrote 60 percent of the programs in the system to prepare for utility industry deregulation. The activities ranged from essentials like billing to helpful refinements like the data on past contacts that telephone agents see as they greet customers.

Fears of corrupting clean code -- along with the need for final testing -- are two key reasons computer users are almost universally talking about freezing all changes in their computers at some point this fall.

"No programmers here are working on Year 2000 right now but nearly everyone is rushing to complete other projects before the freeze," said Ron Kerr, senior manager of information services at Home Depot Inc., which plans to halt changes on Oct. 11. Until sometime next year at the earliest, it may be hard to find a computer worker immune from the gravitational pull of Y2K.



-- Gayla (privacy@please.com), August 31, 1999

Answers

Thanks Gayla,

They're finally starting to "get it". Maybe in a month they'll catch up with the discussions that MCHILDs and a@a.a brought here on IBM's efforts.

The HIPER YR2000 9906-9907 August 1999 discussion is a little fragmented but that's the nature of a developing issue. All the pieces are in the thread started by MCHILDs, my analysis of his effort, and a@a.a's cross post.

If anyone still believes the debunkies, I urge you to retrace the discussion.

The only good news is that there is still a full 122 days to bring the 35,000 OS/390 mainframes up to maintenance and retest the critical applications against the new base operating system.

A full 122 days? We should have started two years ago. We should have been done at Day 500. What do we do now?

-- cory (kiyoinc@ibm.XOUT.net), August 31, 1999.


Cory:

What to do? Start thinking about a 9.5, 10.0.

Start thinking self reliance. Think about bugging out to the Baron's proactively rather than reactively.

If I wasn't such a confirmed agnostic, I'd be spending a lot of time praying.

I think we are in the process of flunking our first species wide technological intelligence test..........

-- Jon Williamson (jwilliamson003@sprintmail.com), August 31, 1999.


Cory, I find it IMPOSSIBLE to believe that IBM would not have sounded an alarm a LONG time ago!!

Why do you suppose they did not??

Ray

-- Ray (ray@totacc.com), August 31, 1999.


Quarterly reports.

-- mommacarestx (harringtondesignX@earthlink.net), August 31, 1999.

Funny, Household International is listed above as having 10,000 programs being replaced by new software. THIS IS NEWS TO ME! I'm a Programmer/Analyst at Household and have never heard this number before. And we are freezing our systems on 9/15, so if we are in the middle of a 10,000 program upgrade, the 9/15 date is shot to hell.

-- Lurker (eye@spy.net), August 31, 1999.


Thanks Gayla,

Looks like 2000 will be the year of "lessons learned." Either that or... "The Year Of Living Dangerously." Or not.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), August 31, 1999.


Link to NY Times article

Chart on what those who are all done are still doing.

Now... what about those who are not "all done".

-- Linda (lwmb@psln.com), August 31, 1999.


Moderation questions? read the FAQ