I only lurk in forums because government employees don't want to be quoted. But in this case, I'll pass on what we've found; maybe it will be useful to you.

I work in the State of Texas Year 2000 project office. Part of my job is to monitor several state agencies and universities. This amounts to high-level IV&V work for the most part. Occasionally we get involved in deeper levels depending on the progress a particular agency was making, and on where we think a problem may be occuring.

But we did monitor state agencies and universities continually for the past two years, so I'm confident that there's not much "spin" in the reports they gave us. in addition, the agencies' internal auditors were often heavily involved, so we had backup for our opinions.

Texas started its project late by most standards. Some agencies and universities began as early as 1995. Most got serious only in late '97. A few started as late as mid-98.

By November 20 (date of the latest report--see below), nearly all of our systems, mission critical or not, were done except for genuflecting to the gods of bookkeeping. By "done" we mean code, hardware, data, telecommunications, etc. has been put back into production. By "bookkeeping" I mean obtaining sign-offs from department heads, sending the results to the project office, and things like that.

You can see numerous reports we've produced about our project at http://www.dir.state.tx.us/y2k/faq/stategov.html According to one bond rating agency, we have more information about our Y2K project than any other state. We did that because the best way to motivate any project is to publicize its progress. And this tactic seems to have worked for us.

We did have some IV&V, mostly sponsored by the Feds. They were very concerned that Medicaid, food stamps, child nutrition, and so on would continue to function. We were uniformly impressed by the rigor with which they performed the IV&V.

One of the universities periodically tested IV&V vendors. None of the vendors lived up to any of the marketing hype. They frequently missed errors that were deliberately introduced into the code. All of them reported errors incorrectly. One vendor's error list was entirely bogus. So we don't have a very high opinion of IV&V at the code level.

The only IV&V that has done us any real good has been when an agency's internal auditors were heavily involved in the remediation project. In a couple of cases they probably made the difference between success and failure. On the whole, I've acquired a huge respect for auditors over the past couple of years.

It looks like we'll spend only 70% to 80% of what we originally budgeted. Much of the "shortfall" (ha! imagine government spending LESS than its budget--is this a great state or what?) is due to overestimating the difficulty of the project. Most agencies did remediation with in-house personnel. Apparently once they got into a regular routine with a set of standard date processing routines and solutions, it took less time than they expected, hence cost less.

We have about 25 OS/390 shops, and most of them set up separate LPARs for testing. None seemed to have the problems Cory writes about. But he does come up with some oddball configurations that state governments are unlikely to have.

After all this $$ has been spent, there is a lot of good news. The best part is that all the big shops now have a complete inventory of all applications, and the best documentation they've ever had. Many now have (new) separate testing environments, new suites of development and testing tools, new configuration management tools, updated operating systems, utilities, compilers, hardware and telecommunications infrastructure. For a few short years everything will be in synch. Keeping it that way is another question, though (the sound of knee slapping and screaming laughter that you hear is coming from mainframers).

Yes, we have had some Y2K problems. One agency forgot to migrate a couple of data files, stuff like that. Early on (1989-1990) we experienced some JAE glitches--about the same time that SSA did. GPS equipment at the Department of Transportation was not a problem and apparently still works well. But on the whole we've had very few glitches compared to what we were expecting, and based on Gartner, Meta, Cutter, and others had predicted.

Embedded systems have been about what Gartner claims. The medical schools discovered nothing that would fail catastrophically in *patient care*. So no one would have died because their ventilator shut down at midnight. The schools did replace about 1.3% of their devices for various reasons, mostly due to billing or logging. Some devices have workarounds such as resetting the date after the rollover. A few *diagnostic* machines had to be replaced because they wouldn't work, though. In one spectacular instance it cost about $70,000 for a new blood analyzer.

So there it is, probably not what some of you wanted to hear. Sorry about that. But nearly every one of the Y2K project leaders in our agencies and universities have the same story. The work got done, and we're experiencing very few residual failures at this point. There are certainly no show stoppers to date.

That's why I don't find it difficult to believe that, on the whole, the Feds are telling us the truth. I don't have trouble believing utilities, either. All of what they're saying matches what we've experienced here.

But we'll know for sure in 15 days. Wishing you the best,

-- Nicholas Osborn (Nicholas.osborn@dir.state.tx.us), December 16, 1999.

-- The Remediator (123@4.com), December 16, 1999

Answers

This was very encouraging. Thank you!

Questions:

Would you rate your shop as above average or average? Sounds to me like its way above average.. Kewl for you.

Have you had the chance to network around with other large shop people, and how would you rate their reported progress in comparison to yours?

Many, many Thanks.

-- art (artwelling@mindspring.com), December 16, 1999.

From your lips to gods ears.

Lets hear a reall hurah for a bitr!!!!

(I am from Missouri, show me)

-- Helium (Heliumavid@yahoo.com), December 16, 1999.

Nicholas, We DO want good news like that. It certainly does cheer me up. BTW, I do know a couple of State of Texas programmers and they are dedicated, hard-working, and very smart people. Thanks a lot for the report!!!

-- Mara (MaraWayne@aol.com), December 16, 1999.

All very encouraging, Nicholas. I hope your results and progress are typical. Thanks

dave

-- dave (wootendave@hotmail.com), December 16, 1999.

Hey, nothing wrong with some good news. Lord knows I can use some.

Thanks for the post.

-- Familyman (prepare@home.com), December 16, 1999.

Awwww...

.....Now this really bums me out! (sarcasm mode off)

-- Patrick (pmchenry@gradall.com), December 16, 1999.

On time and less than budget? Are you sure you're doing govt. work? Maybe this was a dream you had...sounds too much like paradise.

-- TM (mercier7@pdnt.com), December 16, 1999.

Go here and read on about my reply (I'm from Texas BTY) :

This is the original posting of the above...

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=0021cs

-- MMireles (MMireles2@hotmail.com), December 16, 1999.

Dear Nicholas--You sound like a sincere, competent and scrupulous fellow but you are living a fantasy with specific regard to both your own systems and federal claims. Date data formats were found,almost universally, by applying somewhere between 15 and 35 business rules- depending on the skill and dedication of the 'remediator' and the sophistication of the 'tool' used to find dates. The ONLY automated system which I know to a certainty found ALL of the dates applied over 17000 business rules to do so. This system produced over 60M trial lines of remediated code for the feds and after 3 months of checking, double checking and triple checking NO errors were found in the remediated code. NOT ONE. The feds simply didn't want to pay the 35c per line freight and opted not to use this vendor. The Australian govt. in concert with private entities had a sample of 77M lines(of 2 Billion) IV&Vd. The 2B lines had been sent to India and returned. Most of the code(approx 65%) was useless and SATURATED with fatal errors. Again, mammon spoke. Simply decided to save a few bucks and get it done by a high rent version of the hucksters in India who had convinced them that they would get good stuff back at 4c per line.MOST of the remediated code out there is pure dog doody. There is only ONE way that is proven to find ALL of the dates and that method employs and applies over 17000 business rules to find them because that is what is NECESSARY to find them. Try to understand this-with few exceptions, they just didn't want to spend what was necessary to do it properly. Shoddy work by harried, overworked programmers using INCREDIBLY UNSOPHISTICATED tools necessarily produces a bad result. Delude yourself if you must but legacy systems are going down and if you couldn't find the showstoppers in 2 yrs why do you think you'll be able to find them in 2wks-2mos or 2yrs.

-- Get Real (gaf@mindspring.com), December 16, 1999.

Well Nicholas that sounds pretty impressive. If true, this is will be the first time nearly all IT projects finished on time across the board. Very impressive. How on earth did everybody get along with the bureaucrats? I work in the corporate world and I had to do my normal job on top of making y2k changes. Try telling management that we need to freeze new applications. That didn't go very far. I was told "new projects will go on." This hampered my fixes and it was frustrating. I can't be the only one in the world that had that problem. I would imagine that happens practically everywhere.

-- Larry (cobol.programmer@usa.net), December 16, 1999.

Get Real,

Your story of trying to use Indian help on the cheap reminded of a quote I heard during the Gulf War. When all those wells were burning there was a fellow who's job it was to put them out. When asked about his $1,000,000 price tag PER WELL he said "If you think it's expensive to use a professional, just wait until you see how much it costs to use an amatuer". Great line. I'll never forget it!

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), December 16, 1999.

This is so frustrating. Here we have two equally credible reports that completely contradict each other. Both Nicholas and Get Real seem knowledgeable and competent yet they see the same process in two mutually opposing ways. Which one is right? Is the Texas state gov't fully compliant or did they take the el cheapo way out and miss a heck of a lot of dates?

Nicholas is speaking specifically, from his own experience and perceptions. Is Get Real also speaking from specific knowledge of the Texas remediation process or is he speaking generally?

How in the world can the rest of us feel comfortable with our perception of the severity of Y2K with such contradictions popping up all the time?

-- cody (cody@y2ksurvive.com), December 16, 1999.

You can be 100% remediated......the interconnectedness is STILL gonna get ya!....

-- pops (pops@poppin.fresh), December 16, 1999.

Wonderful news, Nicholas. Congratulations.

-- ghost (fading into the@background.com), December 16, 1999.

Yo, Nicholas,

That is GREAT news...look, I have a daughter looking into the University of Texas. She's the Class of 2000. Trust me, the very last thing that I want is for the wheels to fall off.

Would you have any insights as to why the Feds seem to be having so much trouble?? I mean Rep. Steve Horn's last report card had so few successes among "High Impact" Federal Programs, and some IRS Centers had not completed at least some of the inventory. I mean, you are both governmental organizations...how could the Feds get so far out of whack??

-- K. Stevens (kstevens@ It's ALL going away in January.com), December 16, 1999.

To answer Cody, Nicholas is speaking from direct experience with the Texas State remediation. Get Real is speaking theoretically and is making assumptions based on his own paradigm (with a little bit of an arrogant attitude I might add). I prefer to take Nicholas at his word and blow off Get Real who aparently has an emotional investment in the system going down.

-- JoseMiami (caris@prodigy.net), December 16, 1999.

One more thing, I work with the MIAMI-DADE Office of Emergency Management and they are saying the same things as Nicholas, "Problem solved." This is not spin. This is behind closed doors and coming from the guy in charge of the Counties preparations. He has seven days of food set aside and is not even slightly worried.

HOWEVER, I plan to hang on to my food and preps. I am going to watch and see what happens with oil imports. The TEOTWAWKI perspective if fading none the less. Remember what we used to say? "Hope for the best and prepare for the worst" Well, I'm prepared for the worst and my hope is growing that it will be a BITR. Some of us here in this forum (a minority) are "Prepared for the worst and hoping for the worst."

-- JoseMiami (caris@prodigy.net), December 16, 1999.

Hey! All of you children that have nothing to do with working with this problem, shut up! You do not know what you are talking about. Pointing out editorials by GN and others is not knowledge. Unless you you can first hand offer Y2K status, you participate in nothing but gossip. Please don't try and compare your knowledge to someone who is qualified to offer an educated assessment.

-- for real (for@real.com), December 17, 1999.

I reviewed the site and was not too happy. I like to do an end-to- end drill down to evaluate a compliance statement. I could not find the raw reporting information to allow me to do this; plenty of template forms, but no archive of completed forms. There is no read- only access to the DIR tracking system.

Here are some (grr -- hate pdfs, so I will paraphrase a bit) tib-bits gleamed from http://www.dir.state.tx.us/y2k/reports/oct99.pdf:

1. Dept. Public Safety is 98% complete... Only PC/Server Hardware and Software remain to be completed. DPS expects these non-mission critical compliance units to be compliant well before the end of the year.

> Hmmm -- interesting testing if the PC/Servers are not compliant.

2. MHMR is 99% complete. Of MHMR's 27 mission critical compliant units, 26 have been remediated. The one remaining mission critical compliance unit includes nine components.

> I was amused with this one. This is like grouping all the applications (Browser, spreadsheet, word processor, etc) on your PC as one unit. Interesting technique to skew the statistics. If these nine apps were broken out then the compliance falls to 74% complete.

> Some back pedalling then occurs to say 6/9 systems have been implemented (but not deemed compliant) -- then the compliance is 91% complete (guess that is not good enough for the press release).

> Then these mission critical systems are deemed not to have a major impact on the State functions and there are no risks related to these applications -- hmm ok.

3. General Land Office is 98% complete. All mission critical units are complete. The three non-mission critical compliance units that are not complete are: Client Server applications Network Hardware and Software Desktop Hardware and Software None of these systems have a significant impact in the main funtions of the agency.

> Wow, I cannot believe this was put in writing. What a silly statement!

4. The Railroad Commission is 96% complete. Of the 6 mission critical systems, 3 have been remediated. The project Status is defined as 100% Units Tested and 96% Units Implemented.

> One would think that a weighting would bring things closer to 50% complete (3/6 mission critical systems). It is also curious how testing is 100% complete when implementations are not complete -- one would think that testing will have to occur after an implementation.

Well you get the idea -- more silliness is in this report - but my fingers get weary.

Paul

-- Paul (paul@home.com), December 17, 1999.

Yes we used Indians but got cowboys instead.

-- SIRAH (richard.dale@unum.co.uk), December 17, 1999.

I have a problem with the post. We brought P**Soft (fill in the blank trivially) on our campus in 1996. My understanding is that we were the first since the CEO is an alum and cut a deal to break into the University market. There are now dozens of campuses using P***Soft. In any event, the goal was to put in the five major ERP chunks by 2000. While I'm assured that the mission critical stuff has been covered and what's not covered has been dealt with by remediation of older code, only one of five chunks made deadline. Sounds like about 85% missed deadline. Interesting number. I'm assured we're OK, but I would NOT call that a resounding success. I would also be suspicious of huge victories on college campuses based upon the P***Soft rumors that have been travelling around.

-- Dave (aaa@aaa.com), December 17, 1999.

Paul .... I think you found the same old "real story" for Y2K ....

-- good_catch (karlacalif@aol.com), December 17, 1999.

Well, very good! Hope it's as you say............

As a fellow Austinite, you are no doubt concerned with our tech industries here who do quite a bit of trade with Asia. The potential chip bust could make our last oil bust look like a piker.

-- Got Nelda Spears straightened out, too?

-- lisa (lisa@work.now), December 17, 1999.

I'm familiar with the failures of some of the IV&V firms. Arnold Trembley noted the incompetence of some in one of his Time Machine Reports. They sent sample code to a number of firms and only one found the bug intentionally inserted by one of the programmers. They went with that firm. I looked through about 10 of Arnold's reports, but couldn't find the report dealing with this. Anyone interested can review them and verify for themselves.

Master Card

I'm also familiar with some of the results returned from India. Of course Nicholas stated that the work had been done in-house, so that wasn't a problem. Right here in Texas (again) one of the local universities sent some code to India and IDEAL code had been converted to COBOL II. Oops! It took 2 contractors one month of hard work to fix that one.

-- Anita (notgiving@anymore.com), December 17, 1999.

ONCE again the INDEPENDENCE FALLACY rears its ugly head!!!

"If an organization goes off half cocked, without detailed knowledge of how its 'system of systems' works All together...as the vast majority of remediators have done, yet make wholesale changes....they are doomed to failure!!!

From what I understand--and please correct me if I am wrong. It doesnt amount to squat if you are 100% compliant within your specific organization. The minute your system interconnects with another system that has been remediated differently..... well you know the rest.

Please some great brain programmer refute this for me!!! will you!!

-- d----- (dciinc@aol.com), December 17, 1999.

All:

Note that I did NOT say that everything was "compliant." Nothing anywhere is totally compliant. There is no such thing. No one has defined it, no lawyer is going to let his client or company claim compliance because of legal ramifications. It's not going to happen.

Further, I did not say there would be no glitches. Twenty minutes after I posted the reply someone came into my office and told me about an abend caused by a 2000+ date in a purchasing system at an agency that claimed they were finished--testing and all. So we DO expect failures, because we're having them now. No doubt there have already been failures that we haven't heard about. That's probably good--it means they're small enough to be handled like every other glitch. At least up to this point.

The success we've had is due primarily to huge support from the Texas legislature. They're the unsung heroes of this project. They provided funding and encouragement, and put enforcement muscle behind the Y2K Project Office. Every agency and university got the Y2K religion early, to a greater or lesser extent.

That's probably the BIGGEST lesson for all IS projects. If upper-level management believes in the project, provides adequate funding, gets involved early, gets regular and accurate progress reports, the project CAN get done on time and on budget. Our luck with the budget may have been that we overestimated substantially.

(Heavens knows, though, Texas has had its share of late/overbudget/cancelled IS projects)

Paul, Good_Catch:

As Paul points out, you don't have access to source documents. Hence you don't know the basis for the "anomalies" you're noticing. Since I'm the one who maintains the server and designed all the reports you're seeing, you COULD email me (that is a REAL email address) and I'll give you the REAL SKINNY on the numbers. It's a long story, though, so be sure you want to hear the facts before you ask for them.

There are many megabytes of source reports, many file cabinets of paper behind what we say in the summary reports. The Y2K projects, and State agencies in general, are a lot like people. Every one is different. I couldn't make very accurate generalizations about YOU from reading only one of your posts.

Inferences drawn from the summaries are like looking at the tip of an iceberg. You may know that the bulk is far more than you see, but you don't know the SHAPE under the water. So be careful navigating around what you think you see.

Dave:

You're preaching to the choir here. Several agencies intended to use ERP software to "solve" Y2K problems. Advice from experts persuaded us to short-circuit that approach and we persuaded agencies to remediate old code. There were several gut-wrenching decisions like that during the project. Fortunately we had the backing of the Legislature when things like that had to be decided.

I just hope that the ERP software is worth all the agony we go through to install it.

Anita:

We didn't have code sent to India, but did have quite a bit of contract help. In one case an agency sent their code to a US-based remediator and then they went over the returned code by hand line by line. Then they sent it through a time machine. And now they've been in production for a year. The remediator's code was almost bug-free.

So it is (was) possible to get good results, but you definitely had to be careful. The agencies were apparently very cautious about who touched their code, so they got good results. No doubt this is a case of "you get what you pay for, and sometimes considerably less."

Fourteen days and counting, and wishing you all the best especially to the folks in Alabama,

-- Nicholas Osborn (nicholas.osborn@dir.state.tx.us), December 17, 1999.

Let's review this again: we all succeed or fail TOGETHER. Government (including schools) are not wealth producers, they are wealth redistributors. Businesses propel this economy upon which governments and charities are dependent. If the business world, the economic engine, the financial lubrication, bites the big one then all the remediated code in .gov and .org is basicly meaningless...because the tax and donation dollars will not appear..they will disappear.

We all succeed or fail together and that includes ALL those folks who we depend upon overseas as we import huge amounts of oil and electronics and raw materials to feed our economic engine.

There will be success stories and Texas .gov and .org may be among them. But this is no great comfort to me. It must all be done.

-- ..- (dit@dot.dash), December 17, 1999.

myopia reigns supreme

-- d----- (dciinc@aol.com), December 17, 1999.