Woke up to the news on the radio. First thing I hear is that the Food Stamp failure in New Jersey was not due to Y2K after all - it was because of them screwing up trying to put on additional capacity. I'm PBC (posting before coffee) here and have to run. Seems odd to me though that they are changing the story after giving Y2K as the reason right away and then waiting a few days. This failure affected about 180,000 people and got wide attention. I haven't verified anything but perhaps someone else will find a link about this change in their tune. I will try to check in mid-day. Just wanted to give youz guyz a heads up on this, Rob.

-- Rob Michaels (sonofdust@net.com), March 24, 1999

Answers

PR Dept. rule of thumb:

Any computer glitches before Jan. 1 are NOT y2k related (don't want to hurt public confidence or our stock price).

Any computer glitches after Jan. 1 ARE y2k related (and alert the legal dept. to file a lawsuit against whoever is at fault.)

-- rick blaine (y2kazoo@hotmail.com), March 24, 1999.

Rob,

Here's the link to a story on that...

http://nt.excite.com/news/r/990324/02/tech-glitch

[snip]

Food Stamp Glitch In New Jersey Costs Millions

(Last updated 2:56 AM ET March 24)

TRENTON, N.J. (Reuters) - Thousands of New Jersey welfare recipients unknowingly spent $5 million of their April food stamp allocation in March when a computer keying error sent the money out 11 days early, officials said Tuesday.

In what was believed to be the first mistake of its kind in the U.S. food stamp program, about 140,000 families who use electronic debit cards to spend their food stamp money began receiving $23 million in April credits Sunday.

New Jersey officials said the error occurred when an automated file transfer failed because of disk capacity and the transfer had to be performed manually. A state employee keyed in the wrong effective date while transferring files, and the error went unnoticed until Monday.

Officials initially blamed the problem on programming changes meant to make the welfare and food-stamps computer system Y2K compliant. They also overestimated the scope of the problem, saying $30 million in food stamps and 194,000 families had been involved.

"We've been flying by the seat of our pants for the past 24 hours," explained Carol Tencza, a Human Services spokeswoman. "This is not a Y2K story. We had a disk-capacity problem."

[snip]

-- Kevin (mixesmusic@worldnet.att.net), March 24, 1999.

LOL, Rick!

I've been thinking a lot about how pr folks are handling this thing. I've had a running script in my head for weeks now.

Here's it is.

SCENARIO: Major computer problem at a large organization. Effects are public. CEO knows media will swarm like flies on a dead rat. Here's the conversation between the CEO (or V.P of Public Relations) and the Chief Info Tech Officer:

CEO: "What happened?"

IT: "The problem was caused because Y2K (synopsis of answer.)"

CEO: "Was anything else involved?"

IT: "Well, sure. . .there was (names other things that contributed to the problem)

CEO: "So what you're saying is that it wasn't just a Y2K problem?"

IT: (Hesitantly. All eyes in the room boring in on him.) "Well, yes."

Hence follows the press release that something else caused the problem. (Can you hear it now?)

-- FM (Vidprof@aol.com), March 24, 1999.

FM -- ROFLMAO. Rick is right, too.

There are hundreds (probably thousands) of meaningful Y2K failures happening right now and the pace is accelerating weekly. But ANNOUNCING a Y2K falure is the kiss of death and will remain so throughout 1999. Think "legal", among other things.

Aside from the um, corrupting impact of our gov-business culture's endless capacity for institutional deceit which I loathe, I hope we can keep ALL Y2K failures below the noise level of the culture. But we won't be able to ....

Rob, wanna take some bets that when some Fortune 500 companies start going under by 2Q 2000, they're saying, "hey, this wasn't a Y2K problem ....."?

-- BigDog (BigDog@duffer.com), March 24, 1999.

Big Dog,

My prediction is that any obvious failures will be attributed to ANYTHING other than Y2K and it will take a tremendous amount of credible investigative reporting (yeah, right. . .) to prove otherwise. In the meantime--depending on which way the wind is blowing in the board meetings--there will be lawsuits filed. Maybe. I wouldn't count on it. Too public.

After all, if there are major Y2K failures, who's to blame?

The CEO who didn't approve the fixes when he should have.

No company wants to discredit a current or former leader--unless of course--it was a former leader who retired in disgrace.

Let us not forget that we are facing all of these potential problems because top company/organization managers did not see early implementation of Y2K fixes as contributing to profits. With some exceptions, there was clearly no value--in their minds--to their company's bottom line (i.e. stock).

They couldn't see the problem until somebody scared the h--l out of them.

-- FM (vidprof@aol.com), March 24, 1999.

Problems can not be Y2K related for insurance reasons.

Thanks for the good post.

-- Watchful (seethesea@msn.com), March 24, 1999.

The New Jersey welfare failure is Y2K related and the story can't be taken back. Once the media puts it out there, the human mind registers it, keeps it, and believes it. If they come back with a different story no one will believe it. Do you?

-- bardou (bardou@baloney.com), March 24, 1999.

It may very well be that the Food Stamp failure was completely independent of Y2K. But as we have already seen with the Peach Bottom nuclear plant incident in Pennsylvania last month, a problem can easily be triggered by Y2K (testing, in that case), but then later dissected so that it is largely blamed on human error.

People need to recognize that most Y2K problems will not be 100% due to Y2K, but it will be Y2K that gives the "big push". Possible hypothetical examples: Cascading power failures intiated by local equipment that fails due to a Y2K problem; the collapse of the banking system due to very little cash available, but initially caused by people figuring out that all their electronic money may turn to mush on 1/1/2000; etc.

-- Jack (jsprat@eld.net), March 24, 1999.

"Keyed in the wrong effective date"...? So it was date related...nah just a disk capacity problem.

-- Shimrod (shimrod@lycosmail.com), March 24, 1999.

Watchful, you have posted one of the most profound statements I have read in recent days.

Insurance not covering Y2K failures that cause damages will mean that we may well soon live in a weird world of computer failures that no one will blame on Y2K.

I think we're already there. I know of a post office distribution center in a major city that has been experiencing Y2K related failures since January. There has not been one word about it in the media, even though many have noticed our mail delivery has been A LOT slower than usual. No one has publicly connected the dots. I doubt they ever will.

Has anyone experienced strange things with telephone systems at large companies lately? I know of one U.S. Representative whose constituents could not reach her the night before the Clinton impeachment vote because whenever they dialed her number they were told it had been disconnected. She was not a happy camper.

Any similar in your experience? If so--and you bother to call and ask what the problem is, and you get someone talkative (not anyone in management)ask him/her this question (while assuring him/her you won't get him/her into trouble):

"Would I be wrong to assume this has something to do with what your company's doing to fix Y2K computer problems?"

You may be surprised at the answers you receive.

-- FM (vidprof@aol.com), March 24, 1999.

I saw a piece on the local (NYC) news last night. They said the same thing about a computer operator keying in a wrong date. An agency guy said something like "It turns out it wasn't a Y2K problem as we originally thought. However, Y2K is making everybody much more aware of computer problems in general."

To me, what was the most interesting part of the story was the fact that thousands of food stamp users quickly got word of the problem and swamped the stores to spend all that found money. There were interviews and they just couldn't contain their joy at their ability to get down to the store as quickly as possible and have a field day. A clerk in one supermarket was saying that it was a mob. Just makes you wonder how these folks will react under less "happy" circumstances. Now New Jersey needs to decide how to deal with the fact that there may be children going hungry at the end of April because their parents couldn't exercise any self-control...

-- pshannon (pshannon@inch.com), March 24, 1999.

Here's a USA Today/AP account from yesterday about the food stamp bug. I'll leave it to the programmers here to compare it with today's account and decide which is more plausible:

http://www.usatoday.com/life/cyber/tech/cte693.htm

03/23/99- Updated 03:15 PM ET

NJ seeks to recover from Y2K glitch

TRENTON, N.J. (AP)  A Y2K computer glitch gave New Jersey welfare recipients millions of dollars in food assistance nearly two weeks early.

It's the so-called millennium bug's most widespread effect on the public so far, one expert said.

The error affecting 194,000 recipients was introduced Sunday during a test to determine whether a state welfare computer was protected from the problem involving how some computers read dates, said Ed Rogan, a spokesman for the state Department of Human Services.

The error ''made the benefits available as of April 1, 1990, instead of April 1, 1999,'' making next month's benefits available immediately, Rogan said.

Most New Jersey welfare recipients carry benefits cards that work like bank debit cards, so they quickly learned when making purchases that their accounts had grown unexpectedly.

Some called the state to report the error, but others crowded grocery stores to take advantage of the windfall.

All accounts were fixed by Monday afternoon, Rogan said.

The error was introduced when the computer was adjusted two weeks ago to cope with the change to the year 2000. On Sunday, the computer didn't fill in the year when new information was entered. State officials said a worker then tried to fix the omission by typing in ''1999'' but left off the final ''9''  and the computer assumed the blank should be a zero.

''This was somebody who was well-intentioned, trying to help things flow,'' said David C. Heins, director of the state's Division of Family Development.

As much as $58 million may have been involved.

[snip]

-- Kevin (mixesmusic@worldnet.att.net), March 24, 1999.

I know of a post office distribution center in a major city that has been experiencing Y2K related failures since January. There has not been one word about it in the media,

This can also be likened to the credit card Y2K problem that existed for a year. Merchants (including myself) did not have merchant software that could handle exp. dates of 2000 and beyond. We as MERCHANTS were not even notified of the problem as it was being addressed by the banking system during 1998 into 1999.

I was blindsided by this problem when processing the first year 2000 card presented for a payment. No go. That's when I learned of the upgrade to new software.

It is exactly correct to expect business and government to cover up Y2K failures. Insurance bailing on coverage and lawyers chomping at the bit have seen to that. Companies won't even disclose things to their clients until absolutely necessary!

Mr. K
***dislikes credit cards anyway***

-- Mr. Kennedy (who's@the.media?), March 24, 1999.

Tangentially, this story raises another important issue.

Major system upgrades have historically required a lot of time for proper and careful planning and implementation. Even then, there are always lots of wrinkles to iron out. And major upgrades to compliant versions depend on the availability of those versions.

Unfortunately, software vendors also waited until the last minute, and are hit by a double whammy -- they need to get their own houses in order AND remediate their products for their customers.

As a result, necessary upgrades are being postponed until the compliant versions become available (if ever). This in turn leads to a lot of hasty last-minute upgrade efforts, probably late this year.

Are upgrade problems y2k problems? Even if these problems have nothing to do directly with date bugs, y2k has been responsible for forcing the acceleration of a process that should not ever be accelerated. Y2k may not be the proximate cause of many such screwups, but it is certainly an underlying factor in them.

-- Flint (flintc@mindspring.com), March 24, 1999.

The following is from another thread in this forum:

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000e BZ

In a Dec. 28, 1998, memo, Gale told hospital officials that problems with D.C. General's computers were increasing. "The disk drives are too full," he wrote. "This condition is creating an environment that is producing data errors. Such errors are erratic and totally unpredictable. Errors are showing up as missing data and incorrect data. Critical patient results are disappearing."

In an interview, the consultant said that some senior hospital officials  including Chief Executive Officer John Fairman  do not appear to understand the severity of the situation and are moving too slowly to prevent a crisis from occurring later this year. "I don't think that [Fairman] recognizes that the system is dying," Gale said. "Their system can't survive this year."

Now, compare that statement, noting the bold sections, with the following statement in an attempt to disguise the Y2K incident

New Jersey officials said the error occurred when an automated file transfer failed because of disk capacity and the transfer had to be performed manually. A state employee keyed in the wrong effective date while transferring files, and the error went unnoticed until Monday.

[small section snipped]
Article from FOX NEWS

"We've been flying by the seat of our pants for the past 24 hours,'' explained Carol Tencza, a Human Services spokeswoman. ''This is not a Y2K story. We had a disk-capacity problem.''

Looks like a y2koverup to me....IMHO.

Mr. K
***looking at the parallels***

-- Mr. Kennedy (y2kPCfixes@MotivatedSeller.com), March 24, 1999.

We installed our 2000-compliant canned-app patch a week ago, and 40% of our A/P (claims) disqualified, not paid. Big fire.

From somebody who works with eligibility, the wrong-date explanation sounds about right, but I'm curious why they threw out Y2K initially.

-- Lisa (lisa@yeah.right), March 24, 1999.

Thanks for what became an interesting discussion, and a special thanks to Kevin for the links to confirm what I heard early this morning. FWIW, I agree with much of what has been posted here, especially the lawsuit aspect, and the desire to cover a Y2K problem with any other error that is possible to come up with.

One point I haven't seen brought up yet is that silence is working both ways. What I mean by this is that companies do not want any Y2K info coming out - good or bad - with the exception of the carefully crafted BS that is spun to supposedly satisfy the SEC. Remember the "Promises Kept" discussions? Good news that is released and turns out to be wrong later, for example. Silence is what we have been getting, and silence is what will mostly continue. We will get information regarding failures despite this.

BigDog: I'm not the betting kind - LOL. And why as late as 2Q 1999? You're not becoming a PollyDog on us now er ya? :)

-- Rob Michaels (sonofdust@net.com), March 24, 1999.

Hey, Rob --- your turn for a slip: you said 2Q 1999. Unless embedded systems blow (I'm STILL not convinced about that as trivial, though hopeful), it's gonna be death by a thousand "non-Y2K" problems until we pass a certain threshold. Then, rapid devolution, not to Infomagic but to a depression. 2Q 2000 seems about right to me.

BTW, if you want to risk a real email, would enjoy meeting you sometime in Joisey since I get down there regularly.

-- BigDog (BigDog@duffer.com), March 24, 1999.

These explanations are questionable.

How does running out of disc space cause 200,000 benefit checks to be mailed? If anything, the next regular mailing would be short 200,000 benefit checks.

Why does the system require a manually entered effective date? A system performing periodic duties (monthly in this case) is invariably programmed to calcuate the 'next' date. If wrongly caculated, THAT error very well could be a Y2k bug.

Assuming that a manual override of the effective date was necessary, why didn't the edit of this date not catch that 199 (1990?) was not within the domain of valid effective dates. THAT TOO, could have been a Y2k bug.

The claim that the user typed 199 and the system assumed a trailing zero is bogus. If anything, a leading zero is assumed, never a trailing zero.

If they're gonna pull our legs, at least they should come up with more believable stories and quotes. It will become increasingly difficult to hide the truth of Y2k failures. Too many people know too much about systems and software that the BS'ers can cover it all up.

-- Nathan (nospam@all.com), March 24, 1999.

Nathan:

I think a lot of the problem is that technical matters never translate into print very well. Errors like these are almost always do to a sequence of things, and y2k may play a role somewhere in that sequence. It's a question of proximate cause -- just how proximate do you want to get?

It's also a question of definition sometimes. Like asking if the victim was killed by the shooter, the gun, or the bullet? The answer often depends on cases - it's the bullet when we're talking about armor-piercing bullets, it's the gun when we're talking about fully automatic weapons, and it's the shooter if we're talking about murder.

From the various descriptions, this sounds like case where the code was badly remediated. Someone trying a workaround to a date bug invoked an introduced bug, which rattled around in the system and eventually gave birth to a bunch of premature checks somehow.

I can tell you from long experience, the symptoms can be amazingly unrelated to the actual error.

-- Flint (flintc@mindspring.com), March 24, 1999.

BigDog: Ouch, ya got me. I meant 2Q 2000 naturally - duh. You wrote Joisey? OH, now I get it, you mean that there state across the big D where they park in the driveway and drive on the parkway. Still trying to work that one out!?!

Flint: Amen to that last line in your post.

-- Rob Michaels (sonofdust@net.com), March 24, 1999.

Flint,

And I can assure you that the symptoms of a computer "problem" are, without exception, related DIRECTLY to the actual error. Computers perform their tasks with unerring accuracy, following pre-defined logical pathways. The can do little else. Computers do exactly what they are told to do, when we tell them to do it, and they do it blindingly fast. Where the relation of symptom to error may not be readily apparent to us mere mortals, the computer has no such difficulty.

-- Nathan (nospam@all.com), March 24, 1999.

Nathan:

A lot depends on what you mean by 'directly'. With time and perseverence, I can eventually trace the sequence that led from one thing to another. There might be a dozen steps in this sequence. It takes a pretty liberal definition to say the original error that started the chain was the 'direct' cause of the end of that chain. There are an awful lot of contingencies in there.

And in my world, dealing with signal quality and crosstalk issues, the infallibility of the computer is kind of a joke.

But your point is well taken. Just because *we* can't understand how a given error might have led to what happened, doesn't mean that error didn't start the complex chain. After many years, I've lost count of the times the actual error eventually turned out to be something I'd dismissed early on, because it couldn't *possibly* have caused the problem.

Several times a year, we have to spend weeks with a logic analyzer pinning down the *precise* cause of a silicon problem, because the silicon vendor cannot make the error happen in simulation, and *refuses* to believe us without the picture in front of them. And they're always amazed.

So anyone who claims that any description of cause and effect in a computer system is too unlikely to be true, is just showing inexperience. Computers are dumber than people but a lot smarter than programmers.

-- Flint (flintc@mindspring.com), March 24, 1999.

Flint,

Perhaps you originally intended to say something along the lines that symptoms can SEEMINGLY be amazingly unrelated to the actual error. I would readily agree with that.

My point was that the revisionist explanations don't come close to reversing the problem's original attribution to Y2k. Furthermore, some of the explanations could well support other potential aspects of failed Y2k maintenance, testing, or procedures.

We're seeing textbook damage control being applied here. Isn't interesting that few, if any, failures have been (or likely will be) openly admitted to be the result of Y2k issues? You and I know that this is not metrically possible. Once again, the truth is discarded in favor of other, perceptually more valuable ends.

-- Nathan (nospam@all.com), March 24, 1999.

Nathan:

It seems clear that y2k entered into the equation at some point. I believe it will be difficult in many cases for anyone to know the extent to which inaccurate or incomplete remediation contributed to some errors. We really have had errors all along that are not due to y2k, and will continue to have them, y2k or not.

I wrote earlier about errors due to rushed upgrades, even if those upgrades really did cure all date bugs. y2k contributed to such problems indirectly by rushing a slow process. Are these date bugs, really? Depends on how you look at it.

And I'm sure that y2k will be blamed or dismissed as the cause of many errors depending on whether such blame or dismissal seems to be in an organization's best interests at the time. Tracking down the actual cause of problems and trying to factor in the degree to which y2k contributed to the problems might be impossible.

But I don't see any conspiracy here.

-- Flint (flintc@mindspring.com), March 24, 1999.

No, conspiracy is far too strong a word for this particular incident.

It's more like your standard obfuscation with a little don't-admit-to-anything thrown in. If nothing else, this agency has simply reinforced their apparent incompetence. They can't seem to get the story of their own ineptitude straight.

Oh well, it's not like they have any competitors to worry about...

-- Nathan (nospam@all.com), March 25, 1999.