A few experiances at Y2K remediation.

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

I worked as a Y2K programmer contractor for various companies in several different applications during the last three years.

Had it not been for the usual corp politics, Y2K would have been an outstanding success. Confidence is not high.

A few brief examples:

Project 1: I was hired by XBC TV to repair their payroll system. There were 300 COBOL and 200 Easytrieve programs. The contracting firm openend an office in Florida because labor is cheaper there than in the north.

Most of the employees were inexperienced, most just out of school. I was responsible for the Easytrive programs. No one in the group knew Easytrieve, I prepared a quick and dirty training course to cover the basics which lasted about two hours.

It didn't work out too well, I spent weeks going over and over the remediated programs fixing errors.

With all these errors, some very complicated. XBC TV would not allow us to run any test's because they didn't want us to have access to their payroll files. They would do the testing.

I screamed bloody murder and told my employer that without the ability to test, we were turning over junk code to our client.

I was told, thats what the client wants. End of discussion.

After we turned the programs back over to the client, three weeks later XBC came back and said that after testing, there were no errors. That is the biggest bunch of BS that I have ever heard. You can not expand binary dates of different lengths and expect them to line up without some testing.

Project 2: While working for a major credit card company. The plan was to create a parallel remediated system and run a system wide test parallel to the production system. This was a big project. When deep into the project, several of us tried to tell management that there was no way to get all input files in sync for the test.

To make a long story short, management heads were already committed to the project. Finally, after reality set in the project was wittled down and down untill the actual system test was a joke.

I never expected the grid to go down on 01/01/00 and I think that most IT doomers stated that over and over. I don't know what this "I was right and you were wrong" nonsense is all about.

I believe as I have always believed that there is massive database corruption comming soon to a theater near you.

Most companies can probably handle it. The problem is what about the many other's that can't handle it, and what will the effects be on this rediculas bubble stock market.

I put most of my canned foods in boxes today, so I could make room for more.

We are already walking the edge. Confidence is not high.

Cheers,

-- Infidel (Barbarians@thegate.net), January 09, 2000

Answers

One of the things you have to be careful about, when reading any of these first-hand reports, is not to over-generalize. I'm willing to accept Infidel's account at face value, as well as Pliney's corroborating account, because I've seen similar situations myself -- these are examples of the quintessential form of screwed-up projects, whether they're focused on Y2K or anything else.

But obviously, you can't generalize from these two examples and suggest that ALL companies and ALL y2k projects are as bad as these. Indeed, one of the most difficult things for a few of the "higher-level" Y2K consultants -- the folks who were called in to review contingency plans and double-check what the organization was doing -- is that they only saw the best of the best. (It's possible, for example, that that is what caused Peter de Jager to change his opinion about the Y2K outcome).

In my case, for example, it became clear that the companies who were in trouble (which you could usually tell just from the initial conversation with them over the phone) were not going to call in a consultant who would further expose what a mess they were in. Contrariwise, the companies who were in really good shape were the ones who were not only willing, but eager, to call in one, or two, or sometimes three different outside consulting firms to review everything -- just to make sure, double-sure, and triple-sure that they had not overlooked anything. I was briefly involved in one such review situation, where the consequences of non-compliance could have been EXTREMELY catastrophic, and it was one of the most intellectually exhausting experiences I've ever had, being part of a group that was wracking its brains trying to imagine the teensiest thing that might have been overlooked.

It's tempting to over-generalize here, too, and assume that EVERY Fortune-500 company is acting equally responsibly, with equally competent people. But stories like Infidel's and Pliney's remind you that that's not true, either.

My "deja vu all over again" attitude toward Y2K projects would suggest that, in rough terms, 80% of the Y2K projects would be finished more- or-less on time, with more-or-less an acceptable number of bugs; they might or might not be over budget, but that's a secondary issue unless you're a small company with cash-flow problems. I'm willing to believe that Y2K actually got companies scared enough to work harder than before, and try hard to avoid the political nonsense that Infidel and Pliney have described ... which could lead us to imagine a 90% success rate.

But we're being asked to believe that the Y2K success rate is closer to 99%, or 99.9%, and that just doesn't make sense to me -- not only because I've been personally aware of situations like those described by Infidel and Pliney (and not just in small or medium companies, but in very big companies, too), but also because we have this troubling statistic that just won't go away: 30-50% of the SME's (depending on whose survey you feel like believing) did NOTHING to remediate or test their systems. Nada. Zip. Zilch.

As Infidel says, confidence is not high ... at least not amongst the majority of IT managers who have been emailing me since the rollover. The ones who ARE confident seem to be the ones that are working in the great companies that have done great work all along -- before, during and after Y2K. I fear they may have made the mistake of over- generalizing their own good work, and assuming that everyone else is just as competent, well-organized, and disciplined.

Time will tell...

-- Ed Yourdon (ed@yourdon.com), January 09, 2000.


Me,

Remember the old "New Yorker" cartoon, in which two dogs are sitting in front of their master's desktop PC in the home den/office? One dog is saying to the other one, "On the Internet, no one knows you're a dog."

The situation is compounded on this forum, of course, because so many people are using handles and pseudonyms. Gradually, we begin to get a sense of whose postings we can trust, and whose we cannot ... but hardly any of this stuff would stand up in a court of law.

Same thing with the emails I'm getting. In a few cases, I know the individuals involved, know their background, know their corporate situation. But in many cases, I'm getting emails from someone I've not met before, but whose email address is JoeShmoe@bigcompanywithfamiliarname.com ... and the email message starts out by saying, "I'm an IT manager with 15 years of experience, and I've read three of your books, and the experiences I've had with Y2K are ... blah blah blah ..."

There are lots of other Y2K situations I've seen first-hand, as a consultant visiting an organization; like all other Y2K consultants, I've been required to sign a non-disclosure agreement that tends to be more onerous than the normal ones. Hence the combination of legal restrictions and proper business ethics prevents me (and probably many others who post here) from revealing the particulars of the situation.

As for the people who send me emails, I evaluate them pretty much the same way I evaluate the postings on this forum. Infidel's story may be a purely fabricated lie, but it has the "smell" of a truthful statement, and I've seen many of his postings before. I wouldn't bet my family's life on his statement, but I take it as a "plausibly accurate" bit of information, and add it to all of the other flotsam and jetsam floating around in my head...

Don't know if that helps to answer your question...

Ed

-- Ed Yourdon (ed@yourdon.com), January 09, 2000.


Infidel,

Go back and read what I said -- I didn't say that I distrust the honesty of your report. What I said was that on the Internet, no one knows that you're a dog. Even if I knew you personally, Infidel, there is no way that I can guarantee that the various postings on this thread were actually made by you. Depending on the "patterns" that you see associated with postings made by people with a specific handle or pseudonym, you can assign greater or lesser degrees of confidence in the accuracy, credibility, and fundamental honesty of the various postings.

We've considered, on several occasions, making this a closed forum, with "known" user-ids and passwords. Then we could add encrypted signatures, a la PGP and the other mechanisms available. Our hacker friends and the mysterious folks at the CIA and NSA could probably find a way to overwhelm the kind of modest, amateur tools that we have available ... but aside from that, the consensus has always been -- whenever these arguments come up amongst the sysops -- that there is a greater value to anonymity. It protects the lurkers, the informants, the paranoid among us. But it's like being at a masquerade ball in which everone is blindfolded ... nothing is as it seems, and no one is necessarily who you think they are.

Ed

-- Ed Yourdon (ed@yourdon.com), January 10, 2000.


Infidel,

That was one of the most helpful and factual posts I have seen in here for a while. I have been lurking and scanning through the posts and ready to quit. Your info. has encouraged me to stay vigilant. Thanks!

-- JoseMiami (caris@prodigy.net), January 09, 2000.


Exactly Infidel,

Even with testing all the bad code won't
be found. I've had software that worked
perfectly for many hours under testing
only to find it fail when it got to the
field. Users always find a way of doing
things that you never expected.

-- spider (spider0@usa.net), January 09, 2000.



Thanks---this is the kind of report that I've been fearing. I'm a teacher, not a computer expert, and still, to me, "Something" feels wrong. Maybe it's just because after a year and a half of reading, thinking, reasoning about the problems and knowing that great minds were expecting problems, then this Never-never-land that we seem to be in now just seems like the calm before some overdue storm. Logic says: Stay Prepared. Thank you for your excellent post. Ohiomom

-- Ohiomom (Ohiomom@stillprepped.com), January 09, 2000.

This makes me wonder if the tone of Y2K reporting might not change drastically when certain high-paid news department producers and on- screen personalities get royally screwed-up, instead of royally high paychecks. This out to be an interesting one to watch. ;)

WW

-- Wildweasel (vtmldm@epix.net), January 09, 2000.


"I believe as I have always believed that there is massive database corruption comming soon to a theater near you."

Thanks, Infidel! That too is my opinion. I had to hear it from someone directly involved. Hamasaki had been saying it forever. I said in one earlier post, the production systems may be running, but the accountants have yet to be heard from. It's on slow boil.

Thanks for the info from the front lines.

-- W (me@home.now), January 09, 2000.


Thanks for the info. Folks seem nervous; companies "cutting the fat" with layoffs. Hens seem to be spooked, like there's a fox that's been visiting the hen house, just hasn't gone in yet.

-- Hokie (Hokie_@hotmail.com), January 09, 2000.

Let's amplify that concern. Our last project had a database of 1.5 million records of financial transactions. This database has sometimes as many as 10 thousand entries a week. So not big in volume, but large in amount...totally millions. And is a gov't taxation database. Now the problem is the data entry goes from a unix program to a mainframe where hundreds of rules are run against the data until it is scrubbed. And then the data is not considered 'fixed' for four years. But financial decisions and resource management happens as a result. This mainframe is now dead. The rules are something that has evolved over 27 years. These rules were contradictory and seriously confusing, but they had grown in place. As of October, these hundreds of rules had not been replicated. And the data entry program that we wrote for the client had been put aside in that it made the data entry people (all 3 of them) use their computers 'differently'. Admitedly our program was different, but it had the advantage of puting the data into the database (now sql instead of mainframe). So, the client is adding data now? With rules? And now all your financial decisions against this resource are suspect. And potential foder for litigation? Then there is the front end on the resoruce harvesting license side. Their busy time. Confidence is not high. Once again management by the client. Testing was late (Sept, and Oct.) and the client staff (they did not want to pay us for testing) had no testing plan and could not validate their own data. They could not tell if they were see data anomolies or front end bugs. Easy enough to do in the environment, step out of the application and run a sql query to see the results in raw form. Yet, not done. As you say, confidence is not high. Predicting based on their work load, we think that they will have totally screwed up data sets by June. And what cost to scrub? Even if there is no breakdown in their ability to manage the resource. BTW: they still have no data sharing to the feds. Or other gov't agencies. Gonna be an interesting spring.

Planning on taking adding a couple of hives and some more fruit trees here in a bit.



-- pliney the younger (pliney@puget.sound), January 09, 2000.



But, But the grid didn't go down. What you talkin bout Willis?

-- Big Black Cat (Blackcat@bored.com), January 09, 2000.

I am one of those people not technically qualified to have strongly informed positions on what has/hasn't happened and on what might yet happen. Naturally I'm pleased that things have gone so well so far.

A few posts ago, there was a love-fest for Ken Decker who feels there is no reason to stick around. Many others have said there farewells too. Yet there are informed people such as yourself who think the worst is yet to come. Would you, can you offer specifics on what you think is still in store? Do you see utilities holding up or not? If not, why not? Do you foresee "only" a recession and some business failures associated with the data problems you identify? When? How large scale?

Who else agrees with Infidel's pessimissim? What is your basis?

-- Lars (lars@indy.net), January 09, 2000.


Me still waiting. Me still learning. Me will not give up preps. Neither will me add to already extensive preps. Me will speak to *no one* about what may or may not be in the future!

Me know one thing: y2k was not fixed. Not even close.

Either it was a hoax or the worst is yet to come. Time will tell.

-- Me (me@me.me), January 09, 2000.


Everything was too cut and dried, the spin too smooth and the rollover too perfect. Nothing in the real world operates like that.

-- Porky (Porky@in.cellblockD), January 09, 2000.

Yikes, just when I thought it was safe to go in the water!

-- (data@dumpster.diver), January 09, 2000.


Lars,

I do not have eloquence of speech as many poster's here have. I only have my experience at Y2K remediation. From what I have seen, Y2K has been nothing more than the usual CYA nonsense that we all see every day in Corp America.

You would think that this important project would be different, it is about all of us, not our phony balony jobs. We thought wrong. Y2K has been just business as usual in my experience.

From what I can see, the job did not get done. It takes time for data corruption to filter through.

To me the danger is not the loss of electricty, it is the potential piercing of the biggest market bubble in the history of the world.

I believe that Y2K could be the pin that burst the bubble. I suggest you do a little study on the 1929 stock market and make your own educated guess. If not Y2K, then who knows?

We've seen lots of failed Y2K and market predictions. This does make it any less dangerous. I have made my educated guess and have prepared for the worse. Sorry that I can not be more specific.

Protect your family and be prepared for anything for the next three years. The trick is to not purchase anything that you can't use. Then you won't feel like such a fool.

God forbid that we should be made a fool in front of our profound peers.

I personally could care less.

Cheers

-- Infidel (Barbarians@thegate.net), January 09, 2000.


Ed -

You wrote: "As Infidel says, confidence is not high ... at least not amongst the majority of IT managers who have been emailing me since the rollover."

As one might suspect, you have the better vantage point. Not only do you continue receiving input otherwise unavailable to the common folk, but you are qualified to guage, at least somewhat, the credibility of this information.

I take it from your response that you assign a high probability to the appearance of *significant* problems among *some* companies down the road.

Any additional input will be appreciated. A non-technical person such as me can only guess at the meaning of "confidence is not high".

-- Me (me@me.me), January 09, 2000.


I agree with Ken Decker. I still check the site once in a while for some comedy relief. But the issue has been hashed, re-hashed and even corned beef hashed. I haven't seen any new information here in quite a while. Lots of reports of individual problems that may (or may not) be Y2K related - not to mention contrails, aliens, NWO, conspiracy up the yang.

Gary North has already said, "So long and thanks for all the fish." When the King of FUD caves in its probably over. (On the other hand, Gary North was so wrong throughout this event that he's probably got it all wrong again. Maybe he's the contrarian weather vane...Hmmmm. Maybe it says something about the value of a PhD from Cal.circa the late 1960's)

-- Darby (DarbyII@AOL.com), January 09, 2000.


Well Ed,

I also worked for XTE, a phone company. I Remediated one of their small sub systems by myself. I have confidence in that endeavor, but there was no verification of my results. Only reports to the Y2K auditors.

As corp politics go, I found that simple reports to Y2K auditors was all that was required to be declared as Y2K compliant.

As my Y2K experience expanded, I found that the name of the game to be declared as Y2K compliant was because you filed the proper report.

I find it refreshing that you doubt the honesty of my report. A little late, but refreshing.

Cheers,

-- Infidel (Barbarians@thegate.net), January 09, 2000.


Ed -

I guess that's answer enough. It is somewhat shocking the way people use those psuedonyms; that's very offensive to Me.

As I have been saying and as you said, time will tell. I don't need to be a 'polly' or a 'doomer'. My preps are in place and will be consumed/donated over the next year as I (*gradually*) scale back to a 'reasonable' level. I'm keeping the farm on the lake. I love it more as a retreat than as a 'bunker'! I just need to wait and watch.

The truth will out.

-- Me (me@me.me), January 09, 2000.


It take some balls to use your own name on these forums, from the abuse and wacko flaming that goes on. But I have a greater respect for those who do, even if they were wrong about the rollover.

-- Forrest Covington (theforrest@mindspring.com), January 10, 2000.

* * * 20000110 Monday

I agree with Ed! Y2K has transformed into a grand "masquerade," indeed.

Unprecedented, _Unethical_, _Immoral_ non-disclosure statements "ALLOWING" CORPORATE-STATISTS TO _LIE_ (SPIN) TO STOCKHOLDERS AND CITIZENS with vested interests: LIVES and LIVELIHOODS.

I've probably been black-balled out of the "ordinary" IT ranks for my actions of conscience. So be it. The pangs of conscience ain't {sic!} worth the stress, consequences or the "payoff." My payoff: I sleep well with my preparations still intact.

( Ever wonder if there might be a "Y2K black-ball" list somewhere? Why not?! {That's a rhetorical question, folks!} Probably no way to confirm it; although "Red Files" (Michigan!) were "outted"--not purged, though!!--successfully in the 1980's. )

Over all my IT years ( since 1965! ), I have never witnessed ANY program, of any substance--greater than, say, several _hundred_ LOC-- without errors! NEVER, EVER!

I've _seen_ similar Y2K corporate "remediation games" phenomena posted here and elsewhere. It's rampant, I'm sure. Time will tell.

My bona fide credentials (resume) are public domain (verifiable):

< http://briefcase.yahoo.com/bc/rmangus1 >

or directly from:

< http://briefcase.yahoo.com/bc/rmangus1? c&.flabel=fld7&.src=bc&.done=http%3a//briefcase.yahoo.com/bc/rmangus1% 3fa >

At this point--past the immediate horrors of embedded and PLC Y2K failures--naming names (anonamously) of known, pending Y2K cluster- pucks would be the ethically courageous course of valor in the name of humanity.

That'll never happen, or will turn "too late."

GartnerGroup (!), notwithstanding, enough of the production "chain" will "vaporize" ( pseudo-efforts and no effoerts ) to unavoidably impose debilitating--vertically and horizontally--long-term (supplier/consumer/citizen) damage.

Panic? No! ... Vigilance? Indeed! ... Prudence? Amen!

Don't count "Y2K-compliant chickens" before they're "hatched."

Regards, Bob Mangus

* * *

-- Robert Mangus (rmangus1@yahoo.com), January 10, 2000.


So, the question remains - when will the public see visible problems related to Y2K that have serious impacts on their lives (other than nuke shutdowns, loss of surveillance satellites, POS terminals failing, etc.). I'm aware that things are not kosher all over, but the problem still remains in the (way) background at this date. Am I correct in assuming that the first real sign will be at the end of the first quarter?

-- James (jpeet@u.washington.edu), January 10, 2000.

Moderation questions? read the FAQ