Business as usual? The Year 2000 implications of the normal procedures for fixing computer failures.

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

I would like to offer my perspective concerning the view some people have expressed, in that the computer industry deals with glitches and failures all the time, and therefore any failures which the Year 2000 problem may bring will be dealt with as usual and not create hardships.

It is true that computer failures, both hardware and software, occur with regular frequency and are continuously dealt with in normal times. However, I believe that extrapolating those facts to conclude that Y2K problems will have no added impact does not hold true. If one knows or pays attention to the everyday procedures of fixing failures, then a very different conclusion may be reached.

While I recognize that there are plenty of variations in IT departments (these are people we're taking about, after all) there are certain general procedures or steps almost all of them follow, to one degree or another. First, when a business has any of dozens of computer glitches now, often its own employees can figure out what's wrong and fix it. However, despite best efforts, there are also plenty of times when an error message, problem, or failure stumps the in-house IT people. Then one of two steps are taken next.

The first is to call the Technical Support people for the manufacturer of the system, or software, or hardware where the failure appears to be. It's hoped that the more experienced tech support folks can give pointers on how to go about pinpointing what's wrong, or depending on the quality of a technical support team, they may be able to "walk" a person through a fix over the phone.

If tech support can't figure out what's going on either, or they determine the problem is located in another subsidiary system, or the tech support line is so busy it will take a week or more for them to get onto your problem (this often depends on the type of support contract a business has purchased) the next thing that's done is to call up a consulting company. Since some failures can cost a company up to hundreds of thousands of dollars PER HOUR in lost production capacity, if there is any delay at all in getting tech support, the second option of hiring an expert consultant is taken immediately. However, more often all other avenues of repair are taken before money is spent bringing in an expert.

This is the last option because this costs the company big bucks (expert computer consultants don't come cheap) and also, because of human nature, it's sometimes difficult for in-house people to admit the problem has defeated them. However, if the initial problem has reached the point where the company is losing a ton of money because a needed system is down, then getting it fixed is cheaper for them in the long run regardless of what they might have to pay an outside expert, and they realize this.

Also, there are occasions when there is more than one type of failure which happen at about the same and diagnosing this unexpected turn of events can be a real hassle. (Trying to determine what happened when there is both a software glitch AND a hardware failure together, or more than one of each, can be a really tough job. The expectation is usually for one or the other, but not both. Nevertheless, this is a situation my husband has encountered several times in his career.)

Think of what I've just related as similar to the medical field. There are general practitioners who are very capable in an overall way, but if there is a difficult problem they often call up a specialist and talk to them. If that's not sufficient to figure out what's wrong with you, they send you to the specialist himself, or a team of them if multiple health problems turn out to be involved. It's the same basic procedure in computer technology. Those computer systems people which have been around a long time are the ones who have experience with the greatest number and various kinds of failures which can happen, and therefore they have the greatest chance of figuring out what's wrong and how to fix it in a (hopefully) short time frame. (Ladies not in the computer industry may prefer to think of it as sort of like the difference between a new mother who panics because her baby is crying and she doesn't know why, while a grandmother can often take a look and say with confidence, "She's teething." )

Experience is a great teacher. Unfortunately, no one has any experience with a computer problem which has the potential of creating simultaneous failures around the globe, with the added potential that some failures will feed on each other or possibly mask other failures. Also unfortunately, there is a shortage of experienced experts in various computer fields, as well as a shortage of experienced engineers.

Now we come to a crux of the Y2K problem about fixing things after they've failed. If you've ever had to go to a medical specialist, or know someone who has, you might also be aware that very often it takes a long time to get an appointment -- unless it's a life and death emergency and then other appointments are delayed. The most experienced and knowledgeable people are usually very busy. What happens when you have a situation which might cause a huge number of problems -- all at the same time? The normal projections about finding and fixing a problem then go out the window, because it's then unlikely that the "specialists" will be available for everyone at short notice.

This is why I believe it's impractical to assume that the simultaneous nature of potential Year 2000 failures can be dealt with in a "business as usual" fashion. My husband and other consultants I know have experienced plenty of times when various tech support lines have been backlogged during "normal" months, and businesses have been very unhappy to be told "we'll work on your problem as soon as we can, but it's going to be awhile." While doctors have far more episodes of being called out on emergencies, computer people - both in house and consultants - can also get middle of the night calls to "get here ASAP". Or in the case of some consultants, "fly there ASAP".

Those reading who have been in a situation where top management is assembled in a corporate computer center or room, all on the verge of panic, and breathing down the necks of those IT people there -- waiting for someone to diagnose the problem and get things working again, will be less sanguine about the idea of fixing any and all Y2K problems "as usual".

-- Bonnie Camp (bonniec@mail.odyssey.net), May 01, 1999

Answers

Bonnie,

Thanks for the well written "reality check." I expect a lot of us have been thinking along these same lines, it's just that you have put it down in such a straighrforward way. Cory has been warning about these same problems in the big iron arena. Of course, you won't be convincing everyone, and I expect a few of our own disagreeable (those that disagree) Pollys will be telling us why you are close but not quite correct in your presentation, and therefore no big deal.

-- Gordon (gpconnolly@aol.com), May 01, 1999.


Another variation on the business theme, which I noted buried ... deep ... on another thread, ...

... Whats an executive to do if they expect local infrastructure problems? Not just the fix-on-failure kind of internal IT problems. (Although those could be a factor as well).

Would they contemplate something similar to outsourcing, and possibly "park" their mission-critical operations, elsewhere, for "servicing," with a consulting firm or non-competitive ally? All the while, trying to get back on their corporate feet, so-to-speak, on the home turf?

Assume ... in November, companies in a given geographical area KNOW either, their local power utility or telecommunications systems won't be Y2K ready, or are high on the at risk list. But they DO know, one county/state over (or pick a place -- any place), that the probability of infrastructure remaining completely operational, is fairly high. I'd expect, then, to see portions of companies temporarily moving, to keep key business functions operational. Partial cloning, if you will.

Perhaps, it's a variation on the theme of "flight to quality" or even, location, location, location.

Further, in the way of the unexpected, what if, many of the international companies did the same thing. Moved headquarters to another country. Along with key, support personnel, and mission- critical data.

Just something to ponder.

Also ... wondering ... if the stock market holds, could it go through the roof, to say 15,000+ just because organizations "park their funds in the U.S. for the duration?

Some of the expected local/global Y2K repercussions could be completely opposite, to what "logic" holds is possible.

Should certainly be interesting. But with so many variables, being prepared for longer-term durations, is just basic pre-planning wisdom.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), May 01, 1999.


Diane,

Very good speculation, and what with human ingenuity or adaptation, the things you noted are certainly possible. Regarding the stock market, are you factoring in any increase in layoffs and unemployment. Because if we have a meaningful increase in that, it will be because there are serious problems in business and the economy. And if that's the case, there won't be so much money pouring into stocks, and probably less due to people draining down their assets for current needs. Thus it could easily be a buyers market with cheap pricing of stocks and a Dow of 7,000 or less. It all depends on what you think will be the health of the working class and their investment decisions, here as well as overseas.

-- Gordon (gpconnolly@aol.com), May 01, 1999.


Bonnie, you are as always the provider of funtional insight.

I was a consultant for 25 years. Got good enough that I now work for for only those that I choose especially because I get paid on the value of the improvements. I am like an investor of my time. This means I go where I can get a large return of my time investment.

I would ask a firm a lot of questions before taking them on in a crisis. I would want to speak to the top 3 or 4 tech guys who are in the trenches. Are the still working there? Do the still have at least some access to customer support? Have any of their key staff already left? Next, speak to the CEO and fint out the firm's financial status. Its going to be a sellers market.

-- Tomcat (tomcat@cat.com), May 02, 1999.


Gordon,

Im not thinking so much in terms of individual investor monies. Thinking BIG money ... large corporations, large country and state funds, etc.

Given what we think we know about the global monetary IT systems ... at least now (which should be clearer in the fall time frame) ... I just wonder if well see massive transfers to lower risk systems, countries and perceived safe havens ... be they banks or specific stocks. Large amounts of parked capital, could have some interesting side effects for the U.S.

Conversely, economically, and job-wise, I expect to see large layoffs ... domestically and internationally ... increasing as this year continues and well into next year. Sort of a topsy turvy world for an extended period of time. I fully expect to see a wide spectrum of disruptions, in weird and wonderful ways. Everywhere.

The companies (and individuals), et. al., who do well, post-Y2K, will learn to walk on shifting sands and take on flexibility as a daily mantra. Accurate data and information will be essential too. As is knowing who the smart players are. (Not to mention covering all the basics like, power, telecommunications, water, food supply, etc.)

Some later day, Y2K will provide a "hindsight" gold mine for B-school corporate case studies!

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), May 02, 1999.



Another excellent comment on the inadequacy of "business as usual" approaches to dealing with Y2K computer problems was posted by Flint in another thread. Here is a pertinent extract:

"Beyond this, customary bug-handling procedures are often inappropriate. Normally, troubleshooters follow two main lines of attack: First, revert to the backup data and prior software revision. Second, examine any recent code changes. Neither of these standard approaches applies to y2k bugs -- the new code cannot properly interpret the backup date in many cases, the prior code revisions are worse than useless, and the recent code changes are pervasive. Using our traditional methods to combat y2k bugs can be like using a tank to combat poison gas. Doesn't work."

Jerry

-- Jerry B (skeptic76@erols.com), May 02, 1999.


Excellent thread, thank you Bonnie. I noted a lack of disagreement, perhaps some wheels have begun to turn.

-- Will (sibola@hotmail.com), May 02, 1999.

Moderation questions? read the FAQ