Response to R.C. on Oil embedded systems

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

R.C.,

Thanks for your input on the embedded systems/chips resident within the Oil industry.

Some points though:

1. You still didn't answer my question of how many of these inaccesible, irremediable and mission critical systems/chips there are in the world let alone the oil industry. Is it two? Two hundred? Two million? I suspect you and nobody else on this planet knows in which case a rating on the Y2Kometer is not easy to deduce.

2. I wouldn't rate myself a pollyanna though, just more optimisitc (or less pessimistic) than yourself.

3. I don't pretend to have a clue about the oil remediation business. People were asking questions about whether embedded systems in general would cause America (and the UK) to seize up with no power, no heat and no light. It appears even you do not believe that scenario. I don't believe it either, it may be bad news for oil companies somewhere but that doesn't readily translate to Mad Maxdom here in the UK (with 300 years worth of coal underground) or on your side of the pond.

4. You give a scenario from http://www.iea.org/ieay2k/homepage.htm on underwater pipeline emdedded chips. This is what gets me, you place a mission critical chip miles below on the ocean floor and have no contingency to replace it easily. Couldn't they design a system which could be automatically hauled up and remediated on dry land? Why is it that important? Suppose the device polls between 31/12/99 23:59:30 and 01/01/00 00:00:30. The time elapsed comes out as negative (unless it takes the absolute in which case everything is okay) or a very large number and then this forms the divisor for some crazy flow number. The crucial question is what does it do with this number?

If it subtracts the new oil parameter from the pre-2000 one and divides by the huge erroneous time number then the apparent oil flow is near zero. If it just reports the wrong number to the station on the surface - big deal - we know it a Y2Kism and ignore it for get this - when it does the next elapsed time calculation it will be 01/01/00 00:00:30 to 01/01/00 00:01:00 in other words back in sync and no more problems!

By the way, using century/year/month/day/hour/minutes/seconds is a damn expensive way of calculating elapsed time. A lot of chip just count clock ticks - Y2K immune. Do you know how many of these chips count ticks or count centuries (you say some don't even have the source code now so I bet they don't know either!)?

Of course, if the chip decides a zero oil flow rate is a bad thing it may take things into its own hands and shut something down. That of course is not recommended and a workaround such as a manual override or temporarily snipping its failsafe comms link would do this and then reconnect after 01/01/00.

If the chip simply dies forever at least it won't be shutting down things.

5. You may go on about how many myriads of critical chips there are out there. Okay, what I want to know is how many are being fixed or worked around and what the critical mass of failure is. If you don't reach the critical mass then you'll get through. What is the critical mass, R.C.?

6. Regarding your story of guys having to replace whole units which are no longer supported or manufactured. So why can't they shut part of the system down for a few hours to take out the Y2K rollover or reset the date back a few years to give them time? I really need the manuals for these machines to give you a cogent answer.

7. Regarding the Centrilift Downhole monitoring system. Just what does "potential data integrity concern" mean? Is this a mission critical piece of equipment? Will the oil rig or whatever close down if this monitor gets it wrong? Did Centrilift not rate it highly important and just let it slip of the end of their priority fix list in September? Why does it say "potential data integrity concern"? Don't they know whether it will fail or not - or is it this small time window problem I mentioned in point 4 above. Will the downhole monitor automatically shut down the hole? If not, ignore the rollover blip it produces.

The bottom line here is they probably have a new upmarket Y2K compliant hole monitor and they want all you guys to buy it instead. Nothing to do with running out of time I bet.

8. What is the problem with these field acquisition devices, didn't they say they would rollout in August? The rest is all contingency and no panic to me.

9. Non-critical date using chips. Since when were flow documentation for payment transactions critical? Will the refinery shutdown if it gets a payment wrong? Safety - indeed critical - so shut it down over rollover or bar the embedded system's access to the failsafe mechanism (cut the wire over rollover). I say these without knowing the internals of refinery systems - it just sounds like common sense to me having looked at other non-oil systems.

10. Lying PR men. Proof of this to me is: a. In the know oil executives exercising stock options. b. In the know oil people dumping their oil shares. c. In the know oil people taking massive long positions in the crude oil futures market on a hike in oil prices due to reduced productivity.

I presume you predict this too. Is it happening? As for 10% remediation and taking a risk, they must have prioritised the remediation process and done the show stoppers first - logical if they thought time was against them? Tell me, when a unit in a refinery has a "blow-out" does it render the refinery inoperable for good or is it worked around as in so many accidents in the past?

11. I saw the IEE link, it was all mainly fixed, upgraded or replaced stuff.

12. I agree it won't be a Mad Max scenario. You are holding out for a 5 - 6 scenario, I am more in the 3 - 4 I think, not quite Pollyanna or doomer. I think there will also be disruptions and the gov't oil reserves will be important and no doubt allocated where it is most needed. I think things will be bad in third world countries and the economic implications could be severely recessional. I could lose my job but I don't expect to lose my life or limbs. Prepare accordingly.

Regards,

Shuggy.

-- Shuggy (shimei123@yahoo.co.uk), November 02, 1999

Answers

Shuggy,

Why don't you check out the oil links here:

http://www.greenspun.com/bboard/q-and-a-one-category.tcl?topic=TimeBom b%202000%20%28Y2000%29&category=Utilities

A lot of this has already been asked, answered and discussed.

R.C. has spent a lot of time researching, documenting and sharing with all of us on this forum. I personally hope he's taking some time off this thread to enjoy himself rather than answer these nitpicking questions.

I was Ron Quiggins (Shell) lunch partner last year at a Process Control conference where he was speaking. He told me they'd have to shut down the rigs to do full remediation because of safety risks. Couldn't do that though, because then they couldn't generate money and would have a lot of unhappy stockholders.

And I had dinner with the guy from AMOCO [another speaker] who headed their worldwide Y2K deal ... who agreed with what Quiggins said. We talked about North Sea rigs extensively that evening. FYI - They anticipate more problems there than on the Gulf of Mexico. North Sea rigs, and related pipelines, are very complex.

LOL - But, I wasn't wired when these guys told me this stuff. It was "off the record". So believe me or disbelieve me. Quiggins is a nice guy and a VERY good "company" man, who has had a lot of visibility as president of the API.

I'm also the one who gave technical information to a gas transmisison plant in Beaumont, TX in early '98 ... who said they had no technical problems, and were re-assured by their engineering firm that everything was hunky dorey for Y2K. Something I gave him prompted him to do some tests. Guess what? Failure. He called and thanked me for "saving his company". Never charged him a dime. Just came across the business card - "Sabine Gas Transmission Plant". They should be OK. They wouldn't have, if they listened to that engineering firm!

How about my independent oil pals who have companies in the $400-$600 million range ... And one company that went public in '97. LOL - He told me "Don't you worry about it. I heard about that Y2K deal on TV ... and Bill Gates is gonna fix it." This was last October.

Or how about my friend who sold his [oil equipment] "machine shop" for $42 million cash in September '98. New owner (privately-held Austrian company] had no idea about problems on the plant floor when deal went thru. Neither did my friend when he sold it - was happy it wasn't his problem anymore. There's NO way the new owner has this thing worked out!

This nitpicking stuff really bores me. What purpose does it serve this late in the game? It's just an intellectual exercise.

Sorry to be abrupt. A lot of us are tired of this whole deal, have spent a lot of time trying to educate and help others ... and are just plain burned out.

You said: I don't pretend to have a clue about the oil remediation business."

There are many, many more problems in the oil industry than just these technical things you want to talk about.

I decided to leave Houston ... and moved to Oregon. Too many refineries down there for my taste. Local economy still too dependent on oil. I lived thru that recession in mid 80's when home prices plummeted by about 50%. Didn't want to do that again.

-- Cheryl (Transplant@Oregon.com), November 02, 1999.


Cheryl...

Thank you for your in depth explanation from inside the industry. My background in Industrial Process Control includes most industries and utilities except Prtroleum, although I am a Chemical Engineer by training.

The sense I get is that most companies that are behind on their "mission critical" accounting and/or "enterprise" systems are about 2-3 months behind. Therefore the non-embedded computer systems will come in sometime 1st Quarter. The corporations must keep it together with manual work arounds for about 3-4 months in 2000. So called Large Scale Embedded Systems (LSES), which can in some cases sport a Mainboard smaller than a playing card, are the real killers in Y2K.



-- K. Stevens (kstevens@ It's ALL going away in January.com), November 02, 1999.


1. You still didn't answer my question of how many of these inaccesible, irremediable and mission critical systems/chips there are in the world let alone the oil industry. Is it two? Two hundred? Two million? I suspect you and nobody else on this planet knows in which case a rating on the Y2Kometer is not easy to deduce.

My response: They are virtually uncountable. Many are hidden, and previously unknown chips keep popping up. When you reach the magnitude that is reported, you begin to realizethat quibbling in the millions becomes completely "immaterial" and a moot issue. A typical large well will have nearly a thousand systems with embeddeds connected to a typical well. Some cannot be reached, and others can be reached but are sealed and when a box goes out, all one can do is replace it depending on the problem and the reason.

My response: Sometimes items can be worked around by re-designing the system's software configurations...this can take a long time (weeks or months) depending on the severity and complexity of the unit. Not just any old engineer is qualified to do so.

The problem here is... quantity, numbers and the limited qualified personnel capable of solving the problems in the short amount of time given after the problem was first recognized about a year ago. That's right, the magnitude was extremely underestimated and of no real concern by most companies until it was too late to get the job done correctly.

========= You said:

3. I don't pretend to have a clue about the oil remediation business. People were asking questions about whether embedded systems in general would cause America (and the UK) to seize up with no power, no heat and no light. It appears even you do not believe that scenario. I don't believe it either, it may be bad news for oil companies somewhere but that doesn't readily translate to Mad Maxdom here in the UK (with 300 years worth of coal underground) or on your side of the pond.

My Response:

The issue of electricity embeddeds is a far different and far less complex matter than the systems used in the Oil industry. I have chosen to focus rather on the oil industry than the others because I have far more contacts in the oil industry who are on the scene. Coal? How are you going to get it out of the ground? If you use fossil fuel based systems to retrieve the coal, that means there is still an interlinked dependency. Are you guys prepared to get it out of the ground by hand in the quantities needed to replace the missing oil? And if not, how long to get new infrastructure in place? Here in the U.S. where I live, I've not met anyone in 40 years still using coal except for industrial usage such as power plants, and even many of those have been mothballed by our Environmental Protection Agency and the "Clean Air Act." Coal is not a viable alternative for the USA in the short run as a temporary solution to USA energy needsshould the USA lose significant amounts of oil supply.

---------------- You Said:

4. You give a scenario from http://www.iea.org/ieay2k/homepage.htm on underwater pipeline emdedded chips. This is what gets me, you place a mission critical chip miles below on the ocean floor and have no contingency to replace it easily. Couldn't they design a system which could be automatically hauled up and remediated on dry land? Why is it that important? Suppose the device polls between 31/12/99 23:59:30 and 01/01/00 00:00:30. The time elapsed comes out as negative (unless it takes the absolute in which case everything is okay) or a very large number and then this forms the divisor for some crazy flow number. The crucial question is what does it do with this number?

If it subtracts the new oil parameter from the pre-2000 one and divides by the huge erroneous time number then the apparent oil flow is near zero. If it just reports the wrong number to the station on the surface - big deal - we know it a Y2Kism and ignore it for get this - when it does the next elapsed time calculation it will be 01/01/00 00:00:30 to 01/01/00 00:01:00 in other words back in sync and no more problems!

By the way, using century/year/month/day/hour/minutes/seconds is a damn expensive way of calculating elapsed time. A lot of chip just count clock ticks - Y2K immune. Do you know how many of these chips count ticks or count centuries (you say some don't even have the source code now so I bet they don't know either!)?

Of course, if the chip decides a zero oil flow rate is a bad thing it may take things into its own hands and shut something down. That of course is not recommended and a workaround such as a manual override or temporarily snipping its failsafe comms link would do this and then reconnect after 01/01/00.

If the chip simply dies forever at least it won't be shutting down things.

--------------------------

My Response:

Garbage! Things don't work that way in the embeddeds applications in the oil industry.

Go back to the DD thread, and you'll find a lot of your questions already answered. As Cheryl has just posted above you should check out all of the various threads that have already been posted that answers these questions. You question the placement of embedded systems on the ocean floor but there are various aspects to the issue not the least of which is that the oil well is not expected to produce oil forever. It all depends on a given well as to what the lifespan of that well will be. There are other issues and applications besides those involved down the hole. There are also systems involved in stabilizing the platforms that are also not easily accessable.

You also must realize that there were a lot of applications NOT needing a date specificchips but the manufacturer substituted with date chips because they were out of stock and had to get the order out. Seeing how the cost differences were minimal, the bottom line was put it in and finish it, and ship it so we can get our money now andnot wait til next month or next quarter. Hence there are a lot more of these 'animals' out there that are waiting 'ticking' potential time-bomb problems, all unknown to the owner because its in a sealed system that cannot be physically examined for a variety of reasons... including, the fact that to make the examination will recquire an extended unit shutdown that the company cannot afford.

Frankly, you don't understand nor have a clue as to how the oil biz works as is evidenced by your comments and questions now and in the past. At least not the way USA oil companies operate.

====================You said:

5. You may go on about how many myriads of critical chips there are out there. Okay, what I want to know is how many are being fixed or worked around and what the critical mass of failure is. If you don't reach the critical mass then you'll get through. What is the critical mass, R.C.?

My response:

Critical mass can be 1.

I've heard that perhaps less than 5% of the systems have been examined, let alone fixed or tested. There has been no end to end testing done or reported by any oil company (at least that I've seen) or heard. There was one company that had made such a claim but one of my insiders at that company said it was baloney because he did a bunch ofthe work on it and there was NO end to end testing. I don't think that company still has that claim posted on its website. The last time or two I was there, I couldn't find it and the website had been thoroughly revamped.

============ You said:

6. Regarding your story of guys having to replace whole units which are no longer supported or manufactured. So why can't they shut part of the system down for a few hours to take out the Y2K rollover or reset the date back a few years to give them time? I really need the manuals for these machines to give you a cogent answer.

My Response:

These issues have been hashed over repeatedly... "Chad" over on the new DD thread just a few threads below this has posted a succinct explanation. I don't have time to go hunt for my where my comments on this are within the forum. Wells can't be turned on and off and on like a light switch. Shutting down a well to remediate, will usually meana dead well. You lose pressure, you have to drill another well.

=========You said:

7. Regarding the Centrilift Downhole monitoring system. Just what does "potential data integrity concern" mean? Is this a mission critical piece of equipment? Will the oil rig or whatever close down if this monitor gets it wrong? Did Centrilift not rate it highly important and just let it slip of the end of their priority fix list in September? Why does it say "potential data integrity concern"? Don't they know whether it will fail or not - or is it this small time window problem I mentioned in point 4 above. Will the downhole monitor automatically shut down the hole? If not, ignore the rollover blip it produces.

The bottom line here is they probably have a new upmarket Y2K compliant hole monitor and they want all you guys to buy it instead. Nothing to do with running out of time I bet.

My Response:

They are all inter-related and interconnected. You have to also understand there are a lot of failsafes for environmental regulations that are not really 'necessary' except in the political sense. Same is true for OSHA (safety) requirements. These are 2 federal agencies that are out of control at times with needless regulatory restrictions that ham-string the oil industry. So yes, a system may not seem essential but due to gov't regulations, it would be a violation to jerry-rig around it.

========= You said:

8. What is the problem with these field acquisition devices, didn't they say they would rollout in August? The rest is all contingency and no panic to me.

My response:

Problem is that they did NOT roll out in August. The site updates almost daily if not weekly.

=========== You said:

9. Non-critical date using chips. Since when were flow documentation for payment transactions critical? Will the refinery shutdown if it gets a payment wrong? Safety - indeed critical - so shut it down over rollover or bar the embedded system's access to the failsafe mechanism (cut the wire over rollover). I say these without knowing the internals of refinery systems - it just sounds like common sense to me having looked at other non-oil systems.

My response:

There are far more issues involved than just payment transactions, maintaining correct pressures, etc. Again, many date-chips were substituted ... (or at least this has been found to be the case in various limited assessments done in refineries) but were not called for in the schematics. The manufacturer simply subbed them in to avoid creating a lengthy backorder problem.

========== You said:

10. Lying PR men. Proof of this to me is: a. In the know oil executives exercising stock options. b. In the know oil people dumping their oil shares. c. In the know oil people taking massive long positions in the crude oil futures market on a hike in oil prices due to reduced productivity.

My response:

This is a TOTAL NON-STARTER! You really don't know much about America do you???? Have you ever heard of the SEC??? The Securities and Exchange Commission? This would be considered "insider trading" and forbidden by law.

Remember the last oil crisis. The Federal gov't and (I believe some state gov'ts personally "crucified" some oil executives personally. I don't know if they went to jail, but they spent years in the courts in extensive lawsuits. There is now a real fear of a repeat performance...as the oil industry knows that they will be the "whipping boy" for the public and the gov't. They're not about to blatantly break the law in such a manner. They might try to do it under the table but the SEC has been doing a pretty good job of quickly catching folks that do it... too great a risk.

Regarding Oil futures... same situation applies in a matter such as this. The hedging aspect is also heavily regulated. The major oil companies again were hit with very heavy fines and jail time back in the last oil crisis. Nobody wants to do jail time, Shuggy.

+++++=+++ You went on to state:

I presume you predict this too. Is it happening? As for 10% remediation and taking a risk, they must have prioritised the remediation process and done the show stoppers first - logical if they thought time was against them?

All they've been doing is window-dressing Shuggy. Period.

======= You stated:

Tell me, when a unit in a refinery has a "blow-out" does it render the refinery inoperable for good or is it worked around as in so many accidents in the past?

My response:

It depends on the severity. I've seen refineries out for a couple of years or more or back up in about 90 days. Now this, as opposed to a simple fire or a non blow-out.

Multiply that times multiples of refineries and you've got a problem. Shoot, we see situations all the time where if just one refinery goes down at the wrong time it can "spike" the prices dramatically... as much as 20% in a weekend. Imagine just a dozen going down, combined with a LOT of oil wells domestically and massive chunks shut down inside Opec. We're talking significant here. I'm not saying

ALL refineries or ALL large oil wells will go down...but I've had guys in the field tell me that ALL of THEIRS will go down. THEY KNOW IT. The odds are overwhelming or theyknow that the diagrams show bad ones in critical places that when the go, it's a permanent shut down to the well.

=========== You said:

12. I agree it won't be a Mad Max scenario. You are holding out for a 5 - 6 scenario, I am more in the 3 - 4 I think, not quite Pollyanna or doomer. I think there will also be disruptions and the gov't oil reserves will be important and no doubt allocated where it is most needed. I think things will be bad in third world countries and the economic implications could be severely recessional. I could lose my job but I don't expect to lose my life or limbs. Prepare accordingly.

My Response:

IF we get only 24% oil supplies for 6 months... we will run the risk, substantial risk of a Mad Max scenario developing. Especially IF the lights go out... IF the grid goes down, and the phones go down, ... somebody just blow out the candles, 'cause "the party's over."

IF you're a 4 then what are you doing acting like a Pollyanna in your posts?

Frankly, you don't understand the oil business in America, nor understand the processes at all. Which is no big deal if you admit you don't really know. The tone of your prior posts indicates that you think you do know the oil business. You ask nitpicking questions that show only your naivete regarding oil and at times functionon the level of a troll. IMO.

Another point: Cheryl, in her above post refers to Ron from Shell and API... I did see a quote from him in this regard, as well as some others who confirmed this very issue. Maybe the Linkmeister can find it... I don't know if it was on a Shell website or API or in an interview... I'm thinking an interview somewhere.



-- R.C. (racambab@mailcity.com), November 02, 1999.


Hi Cheryl,

Great post, and glad you could take a moment to contribute. Hope things are going well for you in beautiful Oregon. Too bad you haven't had any further contacts in the Oil biz.

-- R.C. (racambab@mailcity.com), November 02, 1999.


You have just witnessed RC in a battle of wits with an unarmed man.Maybe Shuggy should have taken on an easier opponent,like the fast food industry

-- burgermeister (want fries@with.that?), November 02, 1999.


Hi Shuggy,

Thanks the Cheryl and R.C., my comments will be short.

You said: "Of course, if the chip decides a zero oil flow rate is a bad thing it may take things into its own hands and shut something down. That of course is not recommended and a workaround such as a manual override or temporarily snipping its failsafe comms link would do this and then reconnect after 01/01/00.

If the chip simply dies forever at least it won't be shutting down things. "

This is simply wrong. That's not how control systems work.

The little sensors and local microprocessors have to respond to requests and commands from the bigger computers 'up the line.' If the response is missing, or in any way incorrect, the bigger computers will 'decide' that something's wrong and shut down the process for the sake of safety (or so equipment won't be damaged). Even if nothing is happening with the micro, it still has to return a response (this is one type of fail-safe procedure).

Indeed, the tiniest, stand-alone, processors I've designed had a 'watchdog timer' that was separate from the main processor that made sure the main processor was running (if not, the main processor was reset which shut down anything that was happening).

The work-around for many, many of these processes is to reprogram the processor or replace the entire subsystem. There are no wires that can patch around them.

-- Dean -- from (almost) Duh Moines (dtmiller@midiowa.net), November 03, 1999.


We are commercial fishermen,with electronic throttle controls. No manual overrides, when the system fails,you are dead in the water, And these are reasonably accessible, being in the engine room of the boat.Experts are not available and you are on your own. Just imagine how difficult it would be to fix something a half-mile or so down in the ocean! I foresee some rough water ahead. Good sailing to the folks who believe it's going to be a calm jorney into the new year.

-- Eleanore Mameli (eleanore@island.net`), November 03, 1999.

Shuggy you said <<10. Lying PR men. Proof of this to me is: a. In the know oil executives exercising stock options. b. In the know oil people dumping their oil shares. c. In the know oil people taking massive long positions in the crude oil futures market on a hike in oil prices due to reduced productivity. >>

Stock options are long term incentive plans often with limitations and restrictions. But, insiders are allowed to sell their stock positions due to industry fundementals if they so choose. However, if you don't believe in TEOTWAWKI then you keep your options. Also, any exec could short the shares or buy puts on other companies with no disclosure requirements.

As to futures, you can check the open interest on 2000 deliveries anytime you want. Keep in mind that execs buying on their own account would be a fairly small feature in the market. Also, futures are highly leveraged hedging derivatives where the losses can be huge and swift. Small fluctations now could wipe you out today leaving no capital for future oppotunities. I personally believe that the stock market is going to take a sizable hit but have not taken any short positions. Although I'm am waiting for the right opportunity to purchase OEX puts.

-- garyh (garyh@ormutual.com), November 03, 1999.


Moderation questions? read the FAQ