Reference earlier thread:

Deutsche Bank hit by day-long computer shutdown

Note to all those so sure that the only way this was fixed was backing out the changes and reverting to the old software....NOT!

Deutsche Bank software failure

An IBM software upgrade caused a system failure at Deutsche Bank that left it unable to clear international interbank payments on 1 December.

A bank representative said that the failure was caused by a software error. About DM500bn (#167m) of transactions were delayed after the bank upgraded its IBM IMS Transaction Manager software.

He said, "We were advised by IBM to update the system. We tested it throughout November and ran the system live for eight days before it crashed on 1 December.

"A software fix was delivered by IBM and implemented overnight," he added. The failure occurred at a period of maximum system load.

A source with knowledge of the system said it was caused by an error in the OS390 operating system, which then affected the IMS database system and said the upgrade "must have been year 2000-related, otherwise the bank would not have broken its Y2K freeze".

Barry Graham, an analyst at Xephon, said IMS had failed before, and appeared to be peculiarly vulnerable to failure. "It has a limited user base, so problems are not found as quickly as in other databases like DB2.

Deutsche Bank said the problem was not Y2K-related. IBM refused to comment.

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999

Answers

Hoffmeister: Not being a techie, would this incident be indicative of the way US banks have been remediated? Perhaps it was just an isolated case due to local peculiarities?

-- Neil G.Lewis (pnglewis1@yahoo.com), December 09, 1999.

whatever

-- snooze button (alarmclock_2000@yahoo.com), December 09, 1999.

Neil

My guess is it was due to the local setup.

This was what the statement by Barry Graham was alluding to. IMS has a limited user base, so problems due to specific configurations may take a while to develop and surface. Products such as DB2, with a much larger user base, get more specific configurations "tested" by the users, in a much shorter time.

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999.

Why the advised upgrade by IBM?

In a time when systems should be "frozen" because everything is said to be "Y2k ready" this implementationon of new software is suspect given the timing.

Even more so, We tested it throughout November and ran the system live for eight days before it crashed on 1 December.

A simple little crash ON December 1 after running for 8 days? In "Transer Management Software"?

God, Hoff, are you really that dense?

-- Get a clue (or @ get.lost), December 09, 1999.

The most famous last words of western civilization will be a toss up between

Done by 31 Dec 1998 with a year for testing

and

Prepare as if for a 3 day storm

-- a (a@a.a), December 09, 1999.

Let's see now:

1) "Deutsche Bank said the problem was not Y2K-related."

2) "A source with knowledge of the system ... said the upgrade 'must have been year-2000 related, otherwise the bank would not have broken its Y2K freeze."

3) "IBM refused to comment."

Oh, yeah, that clears THAT up! LOL!!!

-- King of Spain (madrid@aol.cum), December 09, 1999.

Get a Clue

Try following your own handle. It's "Transaction", not "Transfer".

Fixes come through for IBM software, and software in general, virtually every day. Do you expect that to stop for Y2k?

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999.

Hoff wrote:

"Do you expect that to stop for Y2k? "

yes you pompous smart ass.

is a "freeze" something you can't quite comprehend? is this concept new to you?

-- Get a clue (or @ get.lost), December 09, 1999.

Hoff, **you** deposit **your** money in Deutsche Bank then.

White Knuckle Time, Hoff. Keep your money where it's safe, hehehe.

-- paul leblanc (bronyaur@gis.net), December 09, 1999.

Hoff, I am speechless. The only thing I can add right now is...

God Bless America



-- the Virginian (1@1.com), December 09, 1999.

At least according to the other thread, the "freeze" at Deutsche Bank didn't start until Dec 1. The article states the upgrade was installed 8 days prior to the "freeze".

For anyone with a "clue", that's worked in IT, "freezes" are relative. One client "froze" production every month for the two- weeks surrounding month-end closing. Didn't mean nothing got in, just more difficult.

My money's at BB&T, and don't see any reason to move it. Thanks all the same.

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999.

If you were a major bank, would YOU want to admit to a major Y2K problem just days before rollover? Because if you admitted it, you would be setting yourself up for the mother of all bank runs! And that run could cascade...

-- Mad Monk (madmonk@hawaiian.net), December 09, 1999.

Hoffy, you state:

1) "At least according to the other thread, the 'freeze' at Deutsche Bank didn't start until Dec 1."

followed by:

2) "The article states the uprade was installed 8 days prior to the 'freeze'."

Regarding statement 1): Can you point me to WHERE on "the other thread" the "freeze" is said to have commenced on December 1? I can't find it.

Regarding 2): The article per se does not say when the "freeze" started; for all it indicates, the freeze could have been effective in November.

-- King of Spain (madrid@aol.cum), December 09, 1999.

Your Highness, it was just a response on the thread. Anonymous, so take it for what it's worth. But he did agree with you.

Dec 1st seems to be pretty popular as a freeze date; so it wouldn't surprise me if it was the truth.

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999.

Isn't Deutsche Bank the largest bank in the world? I guess this would be considered a big account for IBM. What about my little bank down the street? One of 10,000+ in this country. How fast do you think IBM will respond to their problem?

I've been dealing with IBM for over 30 years. When I first got into this business, they had a "will respond to service problems within 2 hours" policy. They would send in a "monkey man" that hooked up a scope to the computer. He looked real busy. But the fact is that he was just holding hands until a real CE showed up. Maybe 8 hours later. Maybe 16.

God Hoff, you sound like a real doomer with this post. You just don't know it. Yea, I know all about your "Y2K is nothing compared to other problems" theory. I know it all too well.

I guess we'll never see any Y2K problems. It will always be blamed on something else. Take your pick, terrorist, squirrel, drunk driver, anything but Y2K.

Just do me one tiny favor Hoff. I have come to like you over the past several months, and wouldn't want to see anything bad happen to you. Just one, small, tiny, favor...

Prepare...

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), December 09, 1999.

Hey Hoffy

Why don't you book an Egypt Air flight out of JFK leaving at 11:50 pm with a stop in Nantucket just off the beach?

I hear they're giving discounts to military types with your intelligence background, you may qualify.

-- Gordon (g_gecko_69@hotmail.com), December 09, 1999.

Yeah, Sysman, I'm sure IBM jumped. No doubt.

Been there. The company I "grew up" at used to have a staff of full- time IBM SE's. That is, until we started replacing the IBM mainframes with Amdahl's.

No, Sysman, I'm sure there have been Y2k problems. Just as sure there will be many more. And I'm just as sure they'll be handled without any form of "collapse".

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999.

Hoffy, as I am sure that you must realize, this is bad, very bad. The article that you cite above is hardly conclusive as to whether "the problem" was or was not Y2K related -- we are getting two different claims, one by the bank's management and one by a source knowledgeable about the actual computer system in question. (IBM is staying neutral.) The "freeze" commencement date issue, regardless of whether it did or did not begin literally on December 1, is quite secondary.

Common sense suggests to me that this major, important central bank experienced a major -- and Y2K related -- disruption of it's services for a significant period of time, which affected many other banks that depend on it, one month prior to January 1, 2000. Common sense suggests to me that the actual Y2K connection is being deliberately downplayed by the bank's management, for fear of what the public reaction might be. Common sense suggests to me that this major, important bank is probably not atypical of many, many other banks.

Common sense suggests to me that we should all be very, very worried about this.

-- King of Spain (madrid@aol.cum), December 09, 1999.

Your Highness

"Common sense" says to me that, whether the upgrade was related to Y2k or not, extrapolating system software implementation problems to date-related processing errors is absurd.

Gordo

Aww, and you were such a big hero of mine. I mean, that speech, "greed is good" and all. Great stuff.

I'll pass on Nantucket. Getting a little too cold this time of year. And Andy never did fork over the Cancun reservations. Thought he could at least share some of those gold profits....

-- Hoffmeister (hoff_meister@my-deja.com), December 09, 1999.

Too bad for Hershey and others that they didn't "freeze" their systems before becoming SAP-suckers.

---KOS

You darn tootin KOS, no dang Bank Pres on up to CEO would let a peep out about y2k causing constipation in the mainframe. The country club benefits would stop, can't have that.

-- snooze button (alarmclock_2000@yahoo.com), December 09, 1999.

Boy Hoff, I sure hope that you're right on this. I really do.

But I don't know. Nobody knows, Isn't that what everybody says these days, nobody knows? So why do you know?

We've had all kinds of stuff show up here in the past few days. Power companies saying that they don't know if they are ready or not. Koskinen, and the NIST report, talking about embedded systems. The ComputerWorld report, "not yet ready." The sad state of small and medium businesses. How much does it take Hoff? How much before YOU question where we really stand?

Since I don't know, I am happy with what I have done to PREPARE. If you, or the popular spin is wrong, no matter, I am ready. And I am happy with my position. Let me say that again, HAPPY!!!!!

Y2K hasn't cost me very much. Some extra food and water, that I can ALWAYS use. A couple extra stacks of firewood for the fireplace, that I can ALWAYS use. A generator, that comes in real handy when the power is knocked out by a thunder or snow storm, that always happens here, that I can ALWAYS use. A big stack of money, that I can ALWAYS put back in the bank if "nothing" happens. Need I go on?

I just don't get it Hoff. Why are you so down on this type of thinking? What is wrong with being PREPARED, for whatever happens in my life?

Let me say this again Hoff. I am ready for Y2K. If it happens, I'm ready. If it doesn't, no big deal. What about you?

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), December 10, 1999.

Little late, snooze.

Just think, you could have "saved" 22,000 others from being "suckers"...

-- Hoffmeister (hoff_meister@my-deja.com), December 10, 1999.

Sysman, I'm HAPPY for you. Truly. People should be prepared for what they think is appropriate.

Never had, nor have, a problem with people who decide to do whatever they see fit.

It's the twisting and distorting of information. It's the over- hyping and over-exaggeration of severity. It's the fear-mongering that I can't handle.

-- Hoffmeister (hoff_meister@my-deja.com), December 10, 1999.

IMS "limited user base"? Excuse me?

The 35 year-old spin doctors may not have a clue what IMS is, but I'd give my right arm for a fraction of a percent of it's maintenance revenue stream.

Let's see if we can do an market approximation... how about 12,000 global IBM MVS (e.g. BIG iron) sites... wild guess would say at least 50% of those have IMS systems.

If one believes the current advertising, DB2 or other relational database engines have REPLACED older DBMS engines like IMS... just not true. When a company has spent a couple of decades building it's core processing around IMS, replacing such a dependable workhorse just doesn't happen... except in the press releases.

Sound bite analysts seem to believe that new software replaces old software... not true... new software is layered on top of old software.

- David

-- David Eddy (deddy@davideddy.com), December 10, 1999.

OK Hoff, I hear ya.

I hope that you don't consider me as being part of the fear-mongering club. I am concerned because of what I have seen in my days as a programmer. I have seen the guts of too many computers, that do run date sensative programs.

Like I said, I hope that you are right. I hope enough time and effort and money has been spent on fixing this problem. I'm just not as sure as you are.

I'ld love to continue this discussion, but I've got an early meeting in NYC tomorrow, and it's almost a 2 hour trip from here.

Carry on Sir Hoff. We'll talk later.

Tick... Tock... <:00=

-- Sysman (y2kboard@yahoo.com), December 10, 1999.

Hoffy: "... extrapolating system software implementation problems to date-related processing errors is absurd."???!!!!!

For the sake of argument, let's ASSUME that the purpose of the upgrade was in fact to allow correct processing in the year 2000. (I.e., we are are going to assume that the source familiar with the bank's system that was quoted in the article is correct.) Therefore, based on this assumption, to NOT install the upgrade would imply incorrect processing in the year 2000.

So, I would think it hardly "absurd" to connect the dot going from a system upgrade to the dot that represents processing of dates correctly. Or, to put it another way: Hoffy, you surely are not trying to limit "Y2K computer problems" to literally the actual run-time processing of year 2000 dates, completely ignoring all the related problems of actually getting to a state of having a complete, working system to perform the processing!!! If an operating system cannot work reliably in the year 2000 (as we are assuming here), then effectively any application it is supporting would also have to be considered unreliable. (That is common sense!!!)

-- King of Spain (madrid@aol.cum), December 10, 1999.

Was the failure Y2K related?

Of course it was! Nobody would be upgrading IMS at this late stage except for Y2K compliance. Sure, the problem itself may not have been Y2K (probably wasn't, since it ain't Y2K yet) but, for my money an upgrade done purely move to a Y2K compliant release almost certainly meets the criteria of Y2K related. Nobody upgrades IMS for fun, we did it earlier this year (from 4.1 to 6.1) and it was many months in the planning, it went in with no problems however. I think a lot of people will have had to do this, the non-compliant rel 4.1 was very popular and stable.

Gotta go now, we just found 400+ assembler modules nobody realised were there, mostly dated in the late 70's. SMF log analysis shows some of the little buggers are still actually running! When we started to dig into them we found clues to 3 JCL lib's we haven't checked either, it'll be a busy weekend!

RonD

-- Ron Davis (rdavis@ozemail.com.au), December 10, 1999.

The IMS 'limited user base' may be irrelevant; some IMS users are key to the world economy. For example: I work for the company which has been doing the remediation for 'Maersk Line', the world's LARGEST container freighting organisation. A large portion of the data on the world's freight (what ship it's on, which container, what's inside the containers, is the freight hazardous, what port is it destined for etc..) is contained in IMS DL/I databases. Fortunately, IMHO, a good job has been done of remediation: this included upgrading to IMS 6, and adding 'windowing' code to process 2-digit dates; testing was completed some time ago. But we wait in hope that 'toast' is not the only commodity to be shipped around the world next year.

-- Risteard Mac Thomais (uachtaran@ireland.com), December 10, 1999.

This is a terrific discussion, thanks everyone.

IMS is gradually being phased out but the IMS install base is huge. Until sysplex, DB2 did not have the power to challenge IMS's performance.

Someone in the thread observed that the failure occurred on December 1. My 000197AF problem hit on December 1, 1979. This is suspecious. I'd like to see the dump and console log for this one.

A non-intrusive way to research this is to pull the November PTF's that mention IMS.

-- cory (kiyoinc@ibm.XOUT.net), December 10, 1999.

David Eddy

Yes, IMS applications have been replaced by DB2, and "other" things (like SA...well, you know). Back in my "employee" days, I believe it was the early 90's when new IMS development stopped.

Maybe "user" base was the wrong term. Definitely far fewer "applications" at this point. Which in essence is the main point; that software such as DB2 gets a much more pervasive user "test" when released.

Your Highness

The point is that, even assuming DB broke their "freeze" (and according to their web-site, as of September, the "freeze" was to begin Nov 1, not Dec 1), the fact is the overall "freeze" will result in far fewer of these implementations than normal. Implementations globally almost without question peaked earlier this year. And whether or not they were due to Y2k is really immaterial.

No, the point is that at least for the coming month or so, the inherent "noise" level will be drastically reduced. That's the whole point of a "freeze". Leaving just the "date-related" processing errors.

Cory

Well, wonders never cease. I actually agree with you here. My first SAP implementation was R/2, and our "experts" insisted on using DB2 instead of IMS, because it was "relational". What a joke. Although it did use DB2, it was basically structured as KEY, then DATA GLOB. Couldn't even use the DB2 utilities to reorg, etc.

Did just see, though, where SAP now has chosen DB2 as their "preferred" database, instead of ORACLE. R/3 actually does utilize a relational underlying databse.

-- Hoffmeister (hoff_meister@my-deja.com), December 10, 1999.

The 'freeze' was for _any_ move/add/change, it started Dec 1. There was a memo sent out via email stating that you would be 'disciplined' if you were caught breaking the freeze. Getting any sort of MAC in within the freeze requires a major approval process and many signatures. My guess (I'm no mainframe guy) is that they (Frankfurt) did whatever they did on the evening of the 30'th go get it in before the Dec 1 date. It's just a guess, maybe Cory is right and it has nothing to do with this. BTW, to get me Y2K aware, I was sent a copy of Ed Yourdon's book by the Y2K committee in 1998. I was already quite aware of the problem from reading Dr. Ed's stuff.

-- me ($$$@$$$.com), December 10, 1999.

A freeze refers to new development and new releases into production. When a company has an emergency fix (software fix stated in the article), do you actually think they won't put the fix into production? Fixes are not subjected to the freeze. Your lowness, get back to mudwrestling.

-- Maria (anon@ymous.com), December 10, 1999.

Hoffmeister,

I have read the article you linked in Computer Weekly. I have read this thread.

I am sorry to say it, but the information presented to us is fragmentary and clearly not complete.

The article states Deutsche Bank said the problem was not Y2K- related. We need to know the following:

(1) Who at DB said the problem was not Y2K-related. (2) Waht is the nature of the particular fix used, and what makes DB say that the fix is not Y2K related. (3) Did IBM tell them that it was not Y2K-related. If they didn't, who in particular concluded that the matter was not Y2K-related?

Perhaps a few phone calls to Deutsche Bank are in order.

You cannot draw a clear conclusion based on the summary presented by Computer Weekly.

I eagerly await more information.

-- Rick (rick7@postmark.net), December 10, 1999.

Bottom line. Those "OCC Blessed" statements of Y2K Readiness issued at the end of June are essentially meaningless if they are still monkeying around with the software and testing it. And I have no reason to believe they aren't...

R.

-- Roland (nottelling@nohwere.com), December 10, 1999.

Sorry Hoffy. Your Cancun tickets are now crude oil call options :o)

Seems to me that this IS y2k related - this is a no-brainer. Freezes are meant to hold off s/w loads. As the annonymous informer said, y2k s/w loads will inevitably bypass the freeze.

I still have ***hopes*** that the financial system will survive as the consequences otherwise will be most unpleasant... this does not bode well, as it will be the death of a thousand cuts just as likeley as data corruption - nobody knows. Just a hunch, for both of us, ultimately.

Almost there. We'll see.

-- Andy (2000EOD@prodigy.net), December 10, 1999.

How come no one mentiomed that this problem took place as soon as the " Look Ahead " date was Jan.1 ???? Eagle

-- Hal Walker (e999eagle@treewwweb.com), December 10, 1999.