California has had one expensive software failure after another, now Gov. Davis is putting a halt to new development until after Y2K.

State Kills Project to Link Welfare Offices

http://www.latimes.com/HOME/NEWS/FRONT/leadstory.html

This link will probably be invalid tomorrow (July 13)

-- kermit (colourmegreen@hotmail.com), July 12, 1999

Answers

Here's the link

-- Linkster (justpractice@linking.com), July 12, 1999.

State Kills Project to Link Welfare Networks Technology: For at least the fifth time this decade, millions are spent on a failed computer effort. The governor sets tighter controls following a quiet decision earlier this year. By VIRGINIA ELLIS, Times Staff Writer

SACRAMENTO--The Davis administration has quietly pulled the plug on another troubled computer project, deciding that an $18-million attempt to link welfare networks had gone so far astray that it would be fruitless to continue spending money on it. A team of consultants and state officials reviewed the project and reported "grave concerns" because it was "operating with few restraints and lacked adequate controls and oversight." Records and interviews show the decision to kill it--an action begun in the waning weeks of the Wilson administration--came earlier this year. Once again, California, the cradle of today's technology, had a humiliating computer failure on its hands. Nobody announced it, but for at least the fifth time in this decade, state government had frittered away millions of dollars trying to construct a new system. This one would have linked four welfare networks. In the last decade, California has produced more--and more expensive--government computer failures than any other state. Nearly half a billion dollars has been wasted on mammoth systems that spun out of control and were abandoned. For taxpayers, the failures have been more than monetary; they have meant lost opportunities to make government more effective and efficient. Thousands of poor children cannot get support money they are entitled to because there is no statewide automated system for tracking absent parents. The wait for a driver's license can be excruciating because the computer system is old and often out of service. The Corrections Department must transport bulky paper files daily as it moves convicts from prison to prison because inmate data are not fully computerized. "This is the home of Silicon Valley; it's so embarrassing," said Sen. Debra Bowen (D-Marina del Rey), the Legislature's leading specialist on computer issues. Mindful of the political ramifications of past wasteful failures, the Davis administration has adopted a tough posture on computer projects, halting most new endeavors until year 2000 problems are solved. Future computer projects, it has said, will have more stringent controls, including closer oversight and earlier involvement by users. "We do not feel that it's appropriate for there to be any failures of any information initiatives or systems on our watch," said Elias Cortez, Davis' chief of information technology. Experts say that when the state gears up to launch new projects again, simple changes in approach can significantly reduce the likelihood of meltdowns. Borrowing lessons learned by the business world, they say the state needs to break projects into smaller chunks, select experienced managers to head them, provide independent eyes and ears--such as outside consultants--to oversee each phase, and involve in every design the people who will actually use the systems. Many of the suggested solutions seem elementary, but all of them pinpoint the mistakes that have led to the costly fiascoes.

Frustrating String of Debacles Ill-prepared for the technological revolution, state government ventured into the Information Age hampered by California's time-consuming contracting process, a lack of expertise--sometimes gross incompetence--in managing or developing computer networks and a naive belief that bigger was better. With no central system for controlling projects, each agency was allowed to proceed helter-skelter. In many instances, the results were disastrous: * In 1994, after spending $51 million, the Department of Motor Vehicles scrapped a six-year effort to modernize driver's license and vehicle registration computers, declaring the new system unworkable. * The state spent $111 million trying to meet federal requirements to establish a statewide network for tracking child support payments before deciding in 1997 to abandon the project. An audit blamed the contractor for delivering a flawed system and the state for mismanagement. * The Department of Corrections began an effort in 1992 to computerize inmate data, but in 1997 terminated an agreement with a private contractor, deciding its design was unacceptable. Although the cancellation delayed essential automation, it also showed that reforms adopted after earlier failures were having some success. This time the contract was stopped early, before millions were spent installing a problematic system. * The California Lottery contracted in 1992 for a system to make it more convenient for players to cash winning Scratcher tickets. A year later, it abruptly canceled the agreement, saying that the contractor had not delivered. Both sides sued, and a settlement cost the state an estimated $52 million. * The project halted early this year, the Statewide Automated Welfare System-Technical Architecture, would have allowed welfare offices in different counties to communicate with one another. Its cancellation pushes the state further behind in plans for a network to grapple with the new demands of welfare reform. Internal documents show that the Health and Welfare Data Center, which led the project, changed its technical direction in mid-course without following state procedures requiring approval from several other agencies. Even with the new approach and the expenditure of millions of dollars, records show, the data center had not produced a workable link.

Adapting to Change Proves Difficult Officials at the Department of Information Technology, which monitors the progress of computer projects, wrote in internal notes that a critical mistake had been a decision to build the system in-house rather than seek an outside contractor with more expertise and experience. The Wilson administration took the first step toward cancellation in December 1998 by refusing to commit any more funds to the project. The Davis administration terminated it. This latest failure bolsters the considerable evidence that innovations by the private sector are not easily transferred to the public sector, where adjustment to radical change is frequently difficult. At the root of California's failures, audits and interviews show, is a government culture that did not adapt easily to the Computer Age. The state's protracted bidding process eats up years of valuable time, making the technology on some projects nearly obsolete before getting off the ground. One welfare network uses a computer language that became outdated during the lengthy contracting and early construction years. The political power of local elected officials discourages consensus on projects that involve different levels of government. The child support system fell apart partly because of disagreements between state officials and local district attorneys, who balked at sacrificing their individual systems to a central design. When the district attorneys disagreed with state decisions, they lobbied Congress for a more favorable outcome. And a general view that technology is best left to "techies" meant that top state managers, including the governor, rarely paid close attention--despite their heavy stake in the success or failure of computer initiatives. Former Gov. George Deukmejian kept his distance from computer projects, as did Pete Wilson, who aides say cared only that he not be embarrassed by their failure. "I think Wilson always sort of viewed [information technology] as being a back-room accounting function that was a pain in the rear and really didn't get you much," Bowen said. Governors throughout the country are only beginning to understand that technology awareness is one of the new skills they need to take to the office, said Larry Singer, president of Public Interest Breakthroughs, a nonprofit group that helps government use computers more effectively. In the states that have been most successful in harnessing computers to the needs of government, the chief technology officer is a member of the governor's cabinet, he said. In California, for many years an office tucked away in a corner of the Department of Finance was the only oversight body. Later, when that office was criticized for being ineffective, the Department of Information Technology was created, although its authority is limited and the legislation that created it provided that it would cease to exist in 2000. In this haphazard structure, there was no one to hold department heads and agency secretaries accountable--no one to demand that each project have a sponsor, a high-level official who would get kudos if it was successful and black marks if it failed.

Years of Delay Before Cutoff An audit determined that millions of dollars were poured into the DMV project long after it was clear that it would never work. But top managers, relying on assurances from technicians that problems could be solved, waited years before stopping the project. "No state, frankly, has botched these projects like California has," Singer said. "It's a leadership problem." California's size and diversity also added to the complexity of projects. Designing a computer system to serve a sparsely populated rural county as well as the city of Los Angeles is, Singer said, "a hell of a strain." John Thomas Flynn, who was Wilson's chief of information technology, said one mind-boggling problem the state confronted with the child support system was trying to design something workable in populous Orange County as well as tiny Alpine County. Besides having to adapt to vastly different demands, computer systems had to be able to cope with huge amounts of data and a high numbers of users. Each of the systems was a pioneering effort because nothing exactly like it had been built. As the most populous state in the nation, California has more of everything--more licensed drivers, more registered vehicles, more divorces, more children owed support money, more prisoners. "Whenever we have tried to use systems from other places, we have gotten burned," said Craig Cornett, director of criminal justice and state administration for the legislative analyst's office, which revealed the state's computer failures. "The scope of our projects is just too big." The state reacted to the size issue by conceiving projects that were mammoth in scope--an approach, said auditors and analysts, that was the single greatest cause of failure because it was exactly the opposite of what was needed. "Large projects fail--period," Flynn said. He said private enterprise learned early that success is much more likely when big projects are broken into parts, with one phase completed before the next one is launched. But complex, time-consuming state contracting laws, Bowen said, discourage officials from dividing projects into components with separate contracts. "In some ways I think [the failures] have been caused by self-inflicted wounds," Bowen said. "We made the [bidding] process so difficult and so lengthy that an agency's natural response . . . is to want to minimize the number of times they have to do it by procuring a gigantic system that will cover everything." Exacerbating the problems of size and diversity was inexperience. With little centralized control, the success or failure of projects often depended on the level of skill within an agency. Few government agencies had managers with expertise in overseeing multimillion-dollar computer projects and their contractors. "We had project managers who had 10 or 15 years of government experience, but they had never managed a major project before," Flynn said. Poor management, state auditors said, was one factor that led to the demise of the child support system. "[Administrators] made numerous management decisions that contributed to the failure of the [project] and cost the taxpayers millions of dollars," the auditors found. But that system, Flynn said, was also dogged by another problem not entirely of the state's making. In order for the federal government to pick up a huge share of the cost, there were strings attached. One federal requirement was that the state copy child support systems that were successful elsewhere. California wasted millions of dollars before the state and federal governments recognized that none of those systems was compatible with a state this large and complex. The tight federal deadlines assumed the existence of successful systems that merely needed updating. But California's statewide system had to be built from scratch. After failing early tests, the defective system was installed in some counties anyway in an effort to meet the federal deadline. "That drives failure: the rush to grab the money," Flynn said.

Competition From Private Sector How could the state that spawned Silicon Valley have a government so slow to catch on with technology? Silicon Valley is part of the reason, Singer said. With its high salaries and insatiable demand for computer whizzes, it sucks up most of California's technology talent. "State government is unable to attract and keep enough highly skilled people," the California Research Bureau, an arm of the State Library, said in a June report. "Salaries are not competitive. . . . Training and advancement opportunities are insufficient." Cortez said that a few weeks ago he lost a program analyst to a local bank that doubled the man's $45,000 state salary. There are a few success stories amid the problems and failures. The Franchise Tax Board, which collects income taxes, built a large, efficient computer system. So did the secretary of state. Both agencies had clear, realistic goals for their projects, experienced managers to run them and tight oversight. They allowed people who would use the system to be involved in the design process. Secretary of State Bill Jones--not a technology sophisticate--was actively involved in each phase of his project, officials said. When the Davis administration launches its projects, Cortez insists, a new attitude toward technology--along with reforms put in place by the Legislature and his predecessors at the Department of Information Technology--will increase the likelihood that these successes will be repeated. He predicts that Davis, formerly the state's controller who in that post ran a department that relied heavily on technology, will be more computer savvy than previous governors. "When I had my 90-plus-minute interview with the governor, I got fastballs, curveballs and anything a person could throw at me. He really questioned me on these issues," Cortez said. Under Flynn, the Department of Information Technology started requiring that projects be broken into phases, that contractors be paid only when they successfully completed a phase and that outside consultants be hired as trouble-shooters to oversee each phase. But those reforms, in the view of the legislative analyst and other experts, do not go far enough. Many of them were added to individual projects and have not become state policy. The hiring of outside consultants, for example, should be mandatory for all major projects, they said. Managers should not be allowed to veer from consultants' recommendations without Department of Information Technology approval. For the child support project, the state paid outside consultants $3 million to oversee the contractor. Yet the consultants' warnings that the contractor had inadequate staff and deficient testing procedures were ignored. One recent report suggested that the standard contract for technology initiatives needs to provide better protection against contractors who do not deliver the goods. The contract for the child support system was so poorly written that it is questionable whether the state can collect damages for the $111-million failure. In the future, they said, more of the risk should be shifted to the contractor, perhaps providing for damages if promised results are not delivered. Sloppy project management should be prevented by a better selection of managers, other experts said. They should have to meet a rigid set of qualifications, including a requirement that their experience be commensurate with the size of the project; someone who has overseen only a $10-million project should not run a $100-million project. And each project should have a godfather--an agency secretary or department head--with direct responsibility for its success or failure. The track records of state agencies are so varied that a system for rating their technological prowess would allow the state to require those with less expertise to get extra help in planning and designing new systems. "The varying abilities of departments to manage computer projects is like the difference between industrialized nations and Third World countries," said one state official. "Some . . . are still keeping their records on index cards." Other experts say the Department of Information Technology's life should be extended and the Legislature should give it more power and prestige. Bowen said the Department of Information Technology should have clear authority to stop projects that go astray and keep unnecessary or poorly conceived ones from ever getting off the drawing boards. It took nearly a year, for example, before the state acted on Flynn's recommendation to terminate the child support project. Cortez has put most of the Department of Information Technology's energies into year 2000 problems. Some legislators worry that it has not devoted enough staff and attention to projects that have been allowed to proceed. Cortez insists that impression is wrong. He has put some of his most experienced people on continuing projects and they are keeping close watch on all of the state's computer initiatives, he said. * * *

Waste and Failure California has spent millions of dollars on computer projects that failed because they were poorly designed and poorly managed. Here are some of the biggest projects to be abandoned: Project: Department of Motor Vehicles Cost: $51 million Project: Child support tracking system Cost: $201 million* Project: Prison information system Cost: $18 million** Project: Welfare system computer linkup Cost: $81 million Project: Lottery Scatcher automation Cost: $52 million * Includes $90 million in federal penalties ** Costs later recovered from contractor

Copyright 1999 Los Angeles Times. All Rights Reserved

-- (@ .), July 12, 1999.

Now we shouldn't jump to conclusions. Everyone knows that it is a whole lot easier to fix two-digit years in a multitude of old systems than to implement one $18 million system. We just know that they have fixed everything.

-- Mr. Adequate (mr@adequate.com), July 12, 1999.

A great article, though probably a lot more meaningful to those who have worked within government. Most of the problem descriptions sound general and hazy unless you've been there and seen it. Creating a government computer system is like chasing a fast-moving target through a minefield of regulations, all of which are interpreted unpredictably, all the while subjected to shifting budgets, under unreachable deadlines.

Much of this nightmare doesn't apply to remediation projects, but all too much of it still does. The sheer incompetence of government programmers is amazing. I've spent literally hours explaining simple things to government programmers that they should know nearly instinctively just to hold their positions according to their job descriptions. I've had to battle past poo-bahs who perform no known function and are just trying to get in the way to be important. Government programmers fear changes in their familiar ruts as much as private sector programmers fear falling into a rut. For government managers, no amount of improved efficiency of public services is worth *any* loss of territory. Change is always the enemy.

Another constant wonder is government lack of any sense of urgency. There's no danger of going broke, there's no competition, there's no profit motive, jobs are secure. My impression is that government programmers have traded low pay for low standards, low pressures, no need to stay current, short hours, and security. You want me to do *what*? Har har, fortunately you can't make me do anything, how long till quitting time anyway? Y2K compiance? Yawn, we've seen these crash projects come and go, nothing every changes, thank God. Time for my coffee break.

At least filing card systems don't have many compliance issues.

-- Flint (flintc@mindspring.com), July 12, 1999.

"We do not feel that it's appropriate for there to be any failures of any information initiatives or systems on our watch," said Elias Cortez, Davis' chief of information technology.

Thats comforting.

At the root of California's failures, audits and interviews show, is a government culture that did not adapt easily to the Computer Age.

Thats not.

"No state, frankly, has botched these projects like California has," Singer said. "It's a leadership problem." 

Thats not either.

Cortez has put most of the Department of Information Technology's energies into year 2000 problems. Some legislators worry that it has not devoted enough staff and attention to projects that have been allowed to proceed.

Its something.

Why do Californians paying Y2K attention, not feel warm fuzzies about our states Y2K status?

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), July 12, 1999.

Yes, an interesting article. But I really think the comments about programmers are uncalled for. In every DP shop, public and private, there may be some people of the sort you describe, but there are generally others who are quite capable and are motivated to do their best--not just for their organization and clients, but because they take pride in doing a good job.

I have worked in both the public and private sectors, and I have seen some projects succeed and others fail. By and large, the failed projects exhibited many serious flaws, but they did not die because of incompetent or apathetic programmers.

This forum apparently had its start on, and is still linked from, Ed Yourdon's home page. Yourdon has been quite critical of the state of American programming over the years, and has worked to help the field mature. But in his writing, and in that of Capers Jones, it is pretty clear that most software disasters are not due to a lack of technical capability but to organizational problems.

In my own experience (over 15 years as a programmer, analyst, and project leader), I would have to say that the most important indicator of a project's potential is whether top management is committed to it.

For a comprehensive treatment of all the reasons why projects fail, read _Assessment and Control of Software Risks_ by Capers Jones.

It is also commonly known that very large projects are inherently risky, and a great number of those in both the public and private sector tend to fail. I don't believe that the failure rate for large projects is any worse in the California state government than in any comparably sized organization. Consider Microsoft, for example: THE success story, at least financially, but they're having very serious problems with NT.

My $.02, Peter

-- Peter Harlan (pharlan@efn.org), July 12, 1999.

Here's a current (and a bit more durable) link to the LA Times archives and that "Welfare Networks Project" story: State Kills Project to Link Welfare Networks

-- Mac (sneak@lurk.hid), July 13, 1999.

Peter:

I'm not trying to say that incompetent or apathetic programmers are the cause of these failures. Poor planning, poor management, lack of cooperation, shifting requirements, etc. are what kills them.

But my experience with government programmers has been much more negative than with private sector programmers. The have tended to fear change rather than welcome it. This may be organizational in nature -- I'd be downtrodden too if I worked in an environment where one is rewarded for mediocrity and punished for excellence. Excellence is conspicuous, a no-no in most government organizations. The same system that protects your job tends to ossify it.

I really don't know if very big, failing projects tend to get killed earlier in the private sector, but I'd be surprised if this weren't the case. Oversight just seems tighter, and the penalty for plodding on with a doomed project more immediate, in the private sector.

-- Flint (flintc@mindspring.com), July 13, 1999.