The following story sent a chill down my spine. Both Colonial and Explorer, the two largest refined products pipelines in the US, now plan to completely shutdown their pipelines New Year's Eve and bring them back up New Years day. One can only suppose that the reason behind their shutting down -- "to guard against any unexpected Y2K incidents" -- is that they are not 100% confident in the ability of their SCADA systems to perform within proper operating parameters over the Y2K transition. Here's the concern, in the testing of many embedded systems the system runs fine through the transition if you DON'T shut it down, but if you power it down, or adjust it in ways that are not standard operating procedures, many times the system does not come back up, or comes back up with unexpected side-effects. If these companies are concerned enough that they feel the need to idle the pipelines, what makes them think they'll be able to bring them back up without glitches?

Curious minds need to know.

P.S.

Both of them are cutting it too close to midnight January 1st UTC (GMT) for my comfort.

--aj

Explorer to Shut Pipe Temporarily New Year's Eve

link

NEW YORK (Reuters) - Explorer Pipeline Co. said Monday it plans to idle its entire oil products pipeline system, which feeds the Midwest, for 20 hours starting New Year's Eve to guard against any unexpected Y2K incidents.

``We will do the same thing as Colonial. We will shut our system temporarily,'' Explorer President and Chief Executive Officer Scott Van Dyke told Reuters by telephone.

The Houston-to-Chicago Explorer Pipeline, the second largest refined products pipeline in America and jointly owned by eight oil companies, can pump as much as 700,000 barrels per day of refined products like gasoline and distillates.

Colonial, the No.1 pipeline firm, has already announced an eight hour shutdown of its entire 5,300-mile-long oil products line from Houston to New York, starting before midnight December 31, to avoid any such disruptions.

Officials at Tulsa, Oklahoma-based Explorer said the Explorer line shutdown will last from 5:00 p.m. (CST) December 31 until 1 p.m. January 1, 2000.

For Colonial, which has a capacity of some 2.1 million bpd, the shutdown is due to start at 6:30 p.m. (EST) and last for eight hours.

Experts fear potential problems to communications or power supply, for instance, due to the so-called Year 2000 computer bug, which may cause some older computers to malfunction after the date roll on December 31, 1999.

``We have excess capacity this time of year so the shutdown will not be a big problem...We should be able to compensate for the supply loss,'' Tom Jensen, Explorer's manager of shipper relations and transportation services, told Reuters.

He said although the Explorer system was Y2K-compliant, like Colonial's, the company was taking no chances.

``If we had any problems with communications or power supply we want to make sure that our line is down at that time so we have no unforeseen problems,'' Jensen told Reuters.

-- Anonymous, November 18, 1999

Answers

AJ,

Not familiar with oil stuff, but don't you think they can probably turn off the oil flow without powering down the control devices? I saw some devices in EU controls work fine on rollover and then have nuisance problem when rolled over with device power supply off - but it required powering down the device, not just turning off the process. Just guessing.

-- Anonymous, November 18, 1999

Cl, I'm sure they can. But if they're not going to do a coordinated full power down and full power up it kind of defeats the purpose, because in that case they are simply postponing this real world test until some point in the future when it happens in uncontrolled circumstances. Another way of looking at is this, what is it they are trying to accomplish by shutting the pipes off? And what does this really mean? Are they going to empty the pipes, or at least depressurize them? Presummably the incident they are worried about is a valve accidentally opening or an over pressure condition. If they simply shut the pipes off and then turn them back on again they haven't prevented anything, they've simply postponed it for a few hours. If they think they are accomplishing something by a simple off/on exercise without a monitored cold start of the control system then they still don't have a full grasp of the nature of the possible embedded systems problems, and that would simply lead me to be even more concerned.

-- Anonymous, November 18, 1999

The same thought ran through my mind when I read these. I recently had an hour long talk with someone in charge of the remediation program at a pretty decent sized utility. He was defending the utility's statement that they had found "no" Y2k problems. He did indicate that they had found equipment that "would not restart" after the new year once shut off (but somehow didn't want to define that as a Y2k problem - go figure). Anyway, there is apparently some equipment out there that will not restart. I personally would rather see it shut down gracefully than not. Presumably they can monitor it carefully when they try to bring it up.

-- Anonymous, November 18, 1999

F. Synder,

Your opinion of devices that fail with power off is correct, but is a larger concern outside of the utility substation/power station, and less of a problem in the utility world.

Commercial applications use a lot of A.C. 120V wall socket power supplies. Equipment in substations is powered by huge banks of DC batteries sized for either 125 Volts DC or 48 Volts DC. This is the ULTIMATE uninterruptable power supply! The incidence of loss of power supply voltage for utility grade equipment is negligible. Besides, the batteries are used to trip/close the big switches/circuit breakers, and a loss of DC battery voltage would make the failure of control devices a moot point.

I found some devices that would (by a strict interpretation) fail for power-off rollover or a power-on rollover/re-boot because they had no internal batter to store the date, and required manual date/time reset after re-boot. This really isn't a problem, because that is the normal procedure today, yesterday and beyond.

-- Anonymous, November 18, 1999

CL,

I'm sorry, but that is the type of comment that is indicative of the ubiquitous laissez-faire engineering these days that truly disturbs me.

Batteries are _not_ the ultimate UPS. Batteries no matter how big will run down. Most battery backed systems in equipment rooms are only designed to last eight hours or so. What happens if the mains are back in nine or ten hours? The battery backed system shuts down, the mains come back, and the little black box doesn't restart because losing the battery backup was never considered a risk.

That's not my idea of a moot point.

-- Anonymous, November 18, 1999

Houston to New York, Houston to Chicago. What is the constant in that sentence? Houston, Houston, Houston. One more time - if Houston doesn't work, the country doesn't work. No/not enough fuel = no electricity. The refineries here in Texas have to work and the pipelines from here to the country have to work. AND, we have to get the oil here from Venezuela and the middle east to begin the whole process. I sincerely hope all the dots work.

-- Anonymous, November 19, 1999

I'd heard that oil in Northern climates can gell if not flowing for an extended period of time. How long would this take?

-- Anonymous, November 19, 1999

AJ,

NO,NO,NO, you missed my point (or I failed to articulate it well enough). DC batteries are not a switched backup - they are the primary and only source in the substation. There are AC powered battery chargers, but the batteries are the source of control device power. The battery system is huge with a high amp-hour capacity. Much beyond what you are imagining in the realm of computer UPS.

-- Anonymous, November 19, 1999

Cl, No, I didn't miss your point.

If you are running off of DC then you need a battery charger. And I am not talking about Jane Q. Public's little 500VA UPS sitting under her desk. What off-the-shelf UPS do you know of that lasts for 8 hours?

I will freely admit that I don't _know_ how big the battery stacks are in an EU equipment room, nor what their amp-hour rating is, nor what their under voltage switch off point is, nor what the total equipment current drain is, nor what their recharge rate is. In other words I have no way of calculating how long those batteries can last without the mains. If you _know_ this information for a _fact_ please share it with the rest of us.

On the other hand I do know how the 48VDC systems work in a telecommunications equipment room. Back in the 1960's and 1970's you could have a power outage on the north-east coast of the US and Canada and your phone would continue to work for days. Now in the late 1990's you're lucky if it works for 8 hours. Until Bell gets the generators out to the smaller Central Offices that is. I can't imagine that the DC supply systems are that much different between the telecommunications industry and the power utilities. Also, the telco's have tended to have a better track record than the power utilities. When was the last time you picked up your phone and didn't hear dial tone? It does take electricity to make that gentle buzzing sound. :-)

On the other hand, PG&E around San Jose, CA here has really been messing with the power lately. For the last few weeks I've been able to _see_ the voltage drop -- "Honey, do the lights seem dim to you?", "Yes, it does seem to be darker" -- poke a voltmeter into the wall and to my surprise see 105.8 volts. No wonder the fridge has been struggling! Sheesh.

-- Anonymous, November 19, 1999