It seems as though we're experiencing a bit of weirdness in the oil-patch. A little extra safety awareness never hurts. Embedded systems, PLC's, SCADA, and such, are usually ubiquitous, easy to forget they're even there. And these systems are everywhere, in almost every industry, chemical, nuclear, oilfield...

December of 1998 I wrote the following, and posted it on csy2k.

ABEND and Big Iron

About 1983, during the dead of winter, ten miles north of the Arctic coastline on a man-made island, a failure in a PLC system almost cost me my life.

At the time I was employed by The Arctic Alaska Drilling Corp., working full time as a lead off floor hand. We were setting up the rig to drill a billion dollar wildcat hole.

This was a difficult rig move, camp was a little over twenty miles from the drill site. From the East Dock of Prudhoe Bay (Prudhoe Bay is where the US gets about twenty percent of its' oil), it was a ten mile drive on an ice road just to get to the pad. That's ten miles driving out onto the Beufort sea.

Instead of everyone working on separate shifts, the decision was made to consolidate the crews, work us fourteen hours and leave two three man crews consisting of a motor-man, forklift operator, and a water truck driver, working opposite twelve's. After the derrick was up we'd move camp and get back to regular the regular schedules.

Moving an Arctic oil rig is no small undertaking, there's usually a couple of very large cranes involved, a fleet of gin trucks or cherry-pickers as they're usually referred to, more than a couple of 966 an 988 caterpillar front end loaders converted to fork-lifts, and in this instance about sixty to eighty hands, all working together in the Arctic night at about twenty to forty degrees below zero, farenheight.

Up there - modular construction techniques are used to split the drilling machine up into "manageable" components. These modules are about sixty feet long - twenty foot wide, made of steel stack'em up, add a sub structure, raise a derrick, and you're in business. Let's make hole.

We'd had some problems earlier in the day trying to raise the derrick, 120 feet long and a couple of hundred tons of iron tower. Something wasn't working right and they had to let her back down onto the stands. Scary stuff.

The motor-man on our crew kind of lucked out, his wife went into labor on the second day of the move, and he went home. I was told "I'd get_to" fill in for him. After changing over from working midnight to noon, then picking up into the new schedule of 6:00 am to 6:00 pm, I got to pull yet another shift, of midnight to noon. Working motors.

Motors power an oil rig, in our case, four V-12 cat's (redundancy), they've generators attached to them, and in a module made of steel, they're very loud. The electricity is run through another module called the SCR room (for silicon rectifiers), magic happens and the juice converts into DC to power the bigger electric motors on the rig. Nice contained system, diesel makes electricity, gets pumped to the oil fired boilers, steam heat is piped around, and it's pretty warm inside, maybe up to sixty degrees.

At the end of the shift my two new compadres, drove back to East Dock to pick up our relief. Well they never came back. Turns out they barely made it back there, an Arctic blizzard came up (aka a "blow"),white-out conditions, fifty-plus mile an hour winds. A blow can go on for an hour, a day, even up to a week. Meanwhile I was in unknowing bliss, finished stuffing insulation, done making my rounds, pretty happy to have relief coming and a warm bed waiting. Decided to go have a donut up off the drilling rig floor in the area known as the dog house, I guess it's called that because you can have a cup of coffee, fill out paper work, or, chase the dog, as it's called - while taking a break.

After an hour I started to wonder "where are those guys?" Started looking around for them thinking they were elsewhere on the rig. I came across a section of insulation I'd just stuffed into place between two modules, except it was laying about ten feet from where it was supposed to be. There was also a couple of feet of snow on the floor.

Oh-oh..

The wind was blowing so hard it had taken the antenna down off the roof of the doghouse. I figured this out after being able to listen the radio chatter, and keying the mike had no affect. Well "OK, say's I", it's warm here, we'll just have to wait it out.

Another hour had passed, time to make the rounds, check the boilers, motors, water tank levels, ect., I heard a terrific banging, it was one of the heavy metal module entrance doors, flapping in the wind. When I went to close the door it almost ripped my arm off and knocked me over. When I looked outside, I saw some of the vehicles that were parked and left running, weren't running any more. That fine snow starts to drift up into the engine compartment eventually making it's way into the carburetor, choking the engine out. A couple of trucks we're still running and so was a forklift, and in that forklift I knew there was a radio.

After making it to the fork-lift, raising East Dock, it was confirmed that I was stuck, cause so was everyone else.

After getting back inside and warming up, I started to get sleepy, just as I began to drift a little, the rig shut itself down.

You might be familiar with the sound of everything coming to a stop when the power goes off. It's a weird kind of non-noise, in this case the howling of the wind replaced the thrum of the engines and whirring of the radiator fans. That's when things started to get a little spooky.

Remember that SCR room I mentioned earlier, seems as though it had some PLC's managing the load from the generators. Something went buggy got unstable and decided to Abend. Up and shut down the engines, all of them, to make matters worse, the compressed air tank was empty (broken gauge displayed full), no way to bring the motors back to life.

The boss on an oil rig is called a tool pusher and he'd told me before I'd started that day, "Howdy, a good motor-man has two items on his person at all times, a flashlight and a crescent wrench," I was lucky I had both. Made my way back out to the 966 which was by then - the last rig running. On the radio with my compadres, it was decided to wake up the boss, (who had loaned me his flashlight).

When his voice came over the radio, he didn't sound.....well...good. He tried to overcome his tone but he didn't quite make it. I have to say I truly liked that guy, "Uh Howdy, don't you go driving off in that there fork lift, OK?"

So, time for a good motor man to go to work and start draining lines, gotta keep those pipes, valves, and radiators from freezing up. Drain that boiler. Don't want all that hardware bursting. When the drain valve was opened up on the boiler the water turned into vapor. The whole inside of the rig to filled up with dense fog and an ice glaze soon started to cover everything. The wind had picked up to shriek level and the groans of all that steel cooling down made for what I can only describe as high drama.

Like a ship going down.

By the time all the lines were drained, I'd started seeing "things" in the fog, lack of sleep probably.

My last trip out to the fork lift was the worst, goggles, refrigiwear suit, down on all fours, get inside, and the interior lights of the cab were a flickering. No more heat in the cab, none on the rig, and what one experiences after exhaustion and massive hits of adrenaline, made for a very calm last conversation over the radio.

Needless to say I did make it through it OK, I never let myself go to sleep, and about eight hours later a really odd looking vehicle called a rollagon, picked me up. Only lost one ball valve to freezing.

OK so what's this got to do with comp.software.year-2000?

My situation - granted, was way, way different than what someone will run into in everyday life, or....is it really so different?

There was a little bit of human error, as in my two compadres who should have waited for relief to show up. A computer systems failure, that darn PLC in the SCR Room. One PLC messing up under a normal situation, what it'll be like in the up-n-coming abnormal situations?

Interdependent systems, relying on each other, to keep the entire system working and alive. This was a closed simple system, with built in redundancy, and all it took to crash it - was one bit of buggy code or a bad chip - in a embedded system.

Lastly, I made out alright in that situation, I kept myself together, maybe being real tired helped. But I wasn't going let myself freeze to death. I started to make plans for a fire to keep warm, added extra clothing, and took precautions.

Just like I'm doing now.

-end-

There's an important factor of the story I needed to summarize in the original post.

Since some of systems today may be disabled, due to y2k work-a-rounds. Manual system back-ups, like the air tank with a broken gauge (reading the gauge - indicated the tank had quite a bit more air - than none), need to be checked on a routine basis. If possible, don't always rely on the analog and digital displayed data, try for a method of positive verification.

While working on a drilling rig, most of us would develop the ability to glean critical information from a myriad set of sources. For instance, one would subconsciously be listening to the sound of the pumps, the vibrations of the floor, the sound of the draw-works, a break in the rhythm, or an odd sound usually meant, something's up. Usually it meant it was time to quit drinking coffee and get back to work, other times it might mean, get out of the way before something heavy falls on you.

Please stay in tune.

Best Regards,

-- Tom McDowell (bullriver@montana.com), January 24, 2000

Answers

hmmm,

I remember reading that when it was first posted.

Thanks, Tom

-- rocky (rknolls@no.spam), January 24, 2000.

Tom,

Medics and Firemen get to know their trucks, truck drivers get to know their trucks, I have gotten to know most of the sedans my company uses (there are a couple new ones I haven't driven enough to get to know but...). we ALL pretty much get to know the background sounds and feels if we work with equipment. NOW we have to tune in to the sounds and vibrations of everyday life, to push that situational awareness that we carry around at work, into the rest of our life. For example, when was the last time any of us walked into our local X-mart and noticed that the stock was down about 40%??? That the racks had been spread out that extra foot, and the shelves were covered, but not deeply????

Situational awareness folks, keeps you alive in "interesting" circumstances (combat, street medicine, fireground, etc.). we are RAPIDLY heading into interesting circumstances.

Chuck

-- Chuck, a night driver (rienzoo@en.com), January 24, 2000.

Tom,

What a great story. Rich with simple answers to the sometimes complicated mysteries of life. Thanks for putting it up.

Gawd, I love this place.

-- semper paratus (still_here_with@my.pals), January 24, 2000.

"While working on a drilling rig, most of us would develop the ability to glean critical information from a myriad set of sources. For instance, one would subconsciously be listening to the sound of the pumps, the vibrations of the floor, the sound of the draw-works, a break in the rhythm, or an odd sound usually meant, something's up. Usually it meant it was time to quit drinking coffee and get back to work, other times it might mean, get out of the way before something heavy falls on you."

you brought a smile to my face on that one Tom. About 20 years ago I worked on a drilling rig out in Elk Hills, CA, was working second tongs... Anyway, when we first set up, some dumbass overflowed the water tank, casing the rig to sink slightly on the west side, so the kelly and elevators hung slightly off center to the west... We were coming out of the hole one day, and I was leaning over the pipe to grap the back of the elevator to pull it back towards me to correct for the drift, when I realized I hadn't heard the brakes going off and the driller was having a coughing fit... the old fart that was the driller liked to free fall till the last minute, then slam down hard right before it was too late..

Man, I was off that floor and half way off the rig by the time it gouged a nice little groove where I HAD been standing... seems the driller started coughing and somehow tangled the counter weight that they used some times when it was easy drilling to keep steady preassure on the brake handle and when he tried to slam down on the brake arm, the cord the counterwieght was tied stopped him short... fortunately, he came down harder and the cord broke, so he kept from demolishing the rig...

Boy, you get to where you know every little sound on those rigs, and if you don't, you usually wind up on disability, if your lucky...

Brought back some memories :)

-- Carl (clilly@goentre.com), January 24, 2000.