All right, here's a weird one that impacted me directly today. I post it here only because, as we've discussed at times, telecommunications is one of the key elements of a properly operating electric system. You might even find this mildly amusing. ;-) I know I didn't - the problem virtually shut down my business all day long. We had to (gasp) find other things to do other than play on the net. ;-)

The following is a general email sent by my ISP late this afternoon to all network subscribers:

Hi everyone,

There are many days when it's fun to go to the office and see what needs to be done. And then there are days like today.

[the remote office manager] called me at 7 AM to let me know the T1 line from [his town]to [our main office] was down, along with several other lines in the [remote] office. He reported a couple problems to Bell Atlantic at 7 AM, and he and I both kept in touch with Bell to see what the status was. Bell kept insisting the problem was caused by a power outage in our office, which was suprizing to the ROM since he was in the office with the lights on, looking at the problem indicators on our equipment.

I got to [the remote office] at noon. Bell arrived about 20 minutes later. The tech seemed a little puzzled by the odd indications from their equipment, so he made a few phone calls. I was in the other room working on something when I overheard him say "You did WHAT??? WHEN??? Well, can you UNDO it?" He came out and calmly asked when the lines went down: around midnight. He started laughing and said the local Bell office did a Y2K software update at midnight. Well, at least we know what caused the problem!

Various Bell people came and went through the office during the afternoon while they tried to figure out what was going on. They seem to have some ideas, but don't have a fix yet.

Maybe the world WILL end on Y2K!

Once the dust settles, we'll add free time to accounts of [the remote office] customers.

Again, this impacted me and my business for a full day today. No billable productivity. Zero. We took care of some housekeeping items that were languishing, but multiply the lost productivity by all of the net dependent businesses in this region...and I'm sure you could quantify this in terms of tens of thousands of dollars.

I've said many times - it isn't the bullet through the heart by Y2k that I'm concerned about - it's the thousand knat bites. My business took one of those bites today.

Rick

-- Anonymous, October 20, 1999

Answers

Very good post Rick, Thanks. Looks like another of the many good examples of Y2K software upgrade problems - fix the y2k bugs, but introduce signficant other types of bugs, sigh.... As I've said before, Y2K is real, it's significant, it just occurred in 1999!.

Regards,

-- Anonymous, October 20, 1999

As I've said before, Y2K is real, it's significant, it just occurred in 1999!

Yardeni and Rubin surveys show more and more big companies are pushing back deadlines. Koskinen says 800,000 small businesses have done nothing; others indicate his calculation is low by an order of magnitude. Many countries are running 12-18 months behind the USA in remediation efforts....

LOL.

Oh, sorry. That's somebody else's line....

-- Anonymous, October 21, 1999

Rick, is this your first direct experience with the Y2K bug? Because my company has had system crashes from the bug, crashes from the updates, and crashes from the server for months now. This week we had our phone lines die, and we had to use our cell phones to let our headoffices know. We've lost weeks of business time, and we can't goto paper and pen when dealing with our business. I've heard that 90% of businesses have already been affected by Y2K. And most of the people I've talked to have said that they have had some problems already as well. But don't worry, I read somewhere that we will all be ready by Dec.31 98, oops, March.31 99, oops, June.31 99, oops.. how about Sept.31, no we've passed that one too. Well certainly we will be ready by Dec.31 99, right?

-- Anonymous, October 21, 1999

Sean - this is the first direct impact, though we've had several inconveniences. Some time back, my business credit card was rejected at a hotel because of the expiration date. The clerk phoned it in, and everything was OK from there. (This was back in 1998; all of my credit cards have 2000 or 2001 expiration dates now, and haven't had any problems since.) No other real problems, at least that I'm aware of.

On a personal level, our bank recently did an ATM Y2k upgrade. The bank had to post a couple of handwritten 3X5 cards on the machine, because after it dispensed money, you couldn't get a receipt or your card back unless you pressed the "cancel" button on the machine. ;-) Quite a few people's ATM cards were eaten that day, AFTER they got the cash, and before the branch manager taped the 3X5 card instructions on the machine. The programming error was apparently fixed overnight, as the machine was back to normal the next day.

But by far, the net outage yesterday was the most disconcerting - I thought Bell Atlantic was *done* with their Y2k upgrades. Hmmmm. Now that I think about it, maybe this was one of those "non-mission critical" upgrades.

BTW - we had contingencies in place for this - and the contingencies didn't work as we would have liked. Only one laptop was able to dial into an out of state number for a national ISP that I use when on the road.

Lastly, here's a link (since my ISP put it on the web) describing the trials and tribulations yesterday...

http://www.vramp.net/happeni ng.htm

-- Anonymous, October 21, 1999

Sean,

I really appreciate your posting on the current news from your own scene, and, I'm no IT expert for sure but I think I see what the problem may be in your own systems. You can take credit for this if you want. Pssssst....tell them to go in and change those dates to June 30, and September 30, and I'll bet it will be off and running again. Silly programmers are always making mistakes like that Some of them didn't even know that there *should* be a February 29, 2000. ;-)

Just kidding around, I hope you know that.

-- Anonymous, October 21, 1999

Hey, is that Lane Core Jr. again??? My own personal, as he calls it "sneering mocker" following up my every post? lol....

Regards,

-- Anonymous, October 21, 1999

Funny that I happened to check in here this morning. I have had one h*ll of a week. Nothing on the scale you discribed, but bad enough.

Some of you may rememeber me. I run an Occupational Safety/Security operation for a large, five hospital healthcare network. Among our responsibilities we provide photo IDs cards for over 6000 employees, physicians, residents, students, consultants, etc. These cards are multi-use. ID, access for doors and parking, and time and attendance.

Well, we upgraded the database this week to make it Y2K compliant. This per the manufacture and with the software they provided. We got one card made, and then it crashed. We lost everything stored in the data base...all photos and information is floating out there in limbo. Thankfully this is a stand alone system, not tied into our network, as I am told it could have corrupted the entire system.

Then....on another data base, the one we store all Security incidents, it too was being upgraded, and it too crashed. Again, all data - for the last 5 years - is lost. The IS folks are working this weekend to see if they can recover any of it.

If the upgrades to make things compliant can cause this, perhaps the cure for Y2K is worse than the illness...or maybe not...

Just my 2 cents...thanks for letting me rant...

-- Anonymous, October 23, 1999

Yardeni and Rubin surveys show more and more big companies are pushing back deadlines. Koskinen says 800,000 small businesses have done nothing; others indicate his calculation is low by an order of magnitude. Many countries are running 12-18 months behind the USA in remediation efforts....

Somebody who claimed that Y2K is a 1999 problem failed to address the three issues I raised.

Gee, what a big whopping surprise.

-- Anonymous, October 23, 1999

Been looking at some of the large telephone company websites and it looks like they are pretty well finished from what I can see. It seems like what we are seeing here though is that just cause something's compliant doesn't mean it still actually works. Oh, so you want it to be both Y2K compliant aannd still run, got it.

-- Anonymous, October 23, 1999

Paul,

Yep, that's the choice they give you. It can be compliant (meaning it's not confused by 2000 anymore) or it can work the same as before, but not both, not yet, maybe in the future, we're not sure, it takes a lot of testing you see, and we don't have time for testing right now. Make up your mind, cause we have umpteen calls we are trying to cover right now. And a bunch of spoil sport disgruntled customers that are suing us. So, what's it gonna be? New software, untested, and take your chances, or keep your old stuff, wait and see what happens, and fix on failure. Make up your mind. You know you really did start awfully late in calling us about this matter! We have our own problems buster. All our new cars and trucks are coming through with a title that says they are a 1900 horseless carriage and the DMV doesn't want to grant a current commercial business tag for something that old. Yada, yada, yada.

-- Anonymous, October 23, 1999

Rick, don't know if it's related, but I spoke to my parents Sunday (10/24) and they said they had a power outage "a few days ago" that began exactly at midnight and lasted exactly six hours. They live in Cherry Hill, NJ. - Judy

-- Anonymous, October 25, 1999

Rick, don't know if it's related, but I spoke to my parents Sunday (10/24) and they said they had a power outage "a few days ago" that began exactly at midnight and lasted exactly six hours. They live in Cherry Hill, NJ. - Judy

-- Anonymous, October 25, 1999