NJ: Possible total system failure of state's new computerized auto emissions testing programgreenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread
Warning lights went off early 01/23/00 By Dunstan McNichol, P. L. Wyckoff and Joe Donohue STAFF WRITER
The press conference was called in a hurry. Gov. Christie Whitman told the reporters gathered in her outer office that she would not rest until she found out how New Jersey's new auto emissions inspection system became such a colossal failure despite warnings along the way.
"If I had known for one second that there were these concerns out there, I would have pulled the plug," she declared on Thursday. But an extensive review of the program's history shows that those conducting the post-mortem won't have far to look. Before and after a $400 million contract was awarded to Parsons Infrastructure and Technology Group to bring home a project that was seriously flawed from the start, plenty of warning lights were blinking.
The signs of trouble were there from the beginning, when the state drew up a request for bids to develop and operate an emissions system, a request that contained such rigid deadlines and unrealistic expectations that all the potential bidders except Parsons Infrastructure backed off.
Then, state officials in a number of departments ignored a consultant's repeated and increasingly strident warnings that the state's 5 million motorists were about to be subjected to horrendous delays and screw-ups. Among those who received the warnings was the state official in charge of the inspection contract.
The consultant, a company hired to monitor Parsons' handling of the project, was right. Seventeen months after the Parsons contract was signed in August of 1998, the state has an auto emissions inspection system that was poorly planned and developed with little true oversight.
Motorists going to get their autos inspected have found they had to wait in line for four or more hours. They have had to get out of their cars to stand around in the cold because no waiting rooms were built. Equipment has failed -- shredding tires, declaring brakes faulty when they are not, and refusing to read crucial data such as vehicle identification numbers. Since the system was launched Dec. 13, as many as half of the 106 inspection lanes at 32 state facilities have been shut down on some days.
Instead of a high-tech solution to a nagging problem, the state's emissions testing program has become a monument to communications breakdowns, slipped deadlines and misgivings chronically ignored. The following account of the auto inspection mess is based on a review of hundreds of state documents, including consultant's memos and e-mail messages, and interviews of dozens of people connected with the project, from men and women working the equipment in the inspection lanes to state officials and company executives. The quest for a new auto emissions inspection program began in 1990, when the federal government amended the Clean Air Act and ordered sweeping measures to clean up air pollution. New Jersey, possessor of the nation's dirtiest air, was put on notice that it would have to develop a tough new system for controlling auto emissions. For years, New Jersey's emissions testing system consisted mainly of an analysis of tailpipe emissions of cars as they idled at 32 state-run DMV stations and at 3,500 private garages. The test was free at state-run stations and, as of last year, cost $25 at licensed private garages.
But the new federal guidelines aimed to test cars under conditions that simulated actual driving conditions. The new tests were to use a machine called the dynamometer -- a pair of rollers embedded in the testing center's floor that allow a car to run without moving, like a jogger on a treadmill.
In its original Clean Air Act regulations, published in 1992, the Environmental Protection Agency insisted that states use a dynamometer that spun wheels at 50 mph for four minutes. Faced with accidents and long delays at testing stations that led to motorist revolts in Colorado and Texas, the EPA in 1995 let New Jersey and other states opt for a dynamometer test of 50 seconds, with wheel speeds of only 15 mph.
And the EPA, impatient with a series of state delays, initially gave New Jersey until 1997 to change to the new testing system. Testing companies, attracted by a potential contract worth hundreds of millions of dollars, began lining up New Jersey lobbyists and financing local political campaigns to help get a piece of the action.
But when the bids came due in 1997, only one company, a tiny outfit based in Israel with no U.S. emissions testing experience, made a pitch for the work.
Envirotest and Keating Technologies, among the leading auto inspection companies that backed away from the first deal, say now they found the bid specifications too complicated and unwieldy. The Israeli firm's offer was declined, but by then the state had lost time. With only two years remaining before expensive federal sanctions kicked in for not meeting the clean-air deadline, the state had to draw up a new set of bids and persuade a contractor to set up the new system in record time.
Following a conference with prospective bidders in late 1997, the state made several changes in its bid proposal. It also added a rigid set of performance deadlines designed to put the new program on line by Dec. 13, 1999. In the problem-plagued months to come, this deadline became nearly Holy Writ in the minds of nervous officials: Unless it was met, they feared, as much as $1 billion in federal highway funds could be lost.
At least 300 private garages, for instance, would have to be hooked into a new computer system by Oct. 1, 1998 -- just 90 days after the date for winning the bid, the proposal said. And the successful bidder had to commit to a preliminary deadline by which it would have at least one-third of the new inspection system ready to begin offering the new emissions test.
A contractor who missed a deadline, bid documents warned, would be fined $33,334 a day until it was met.
"These time periods will become critical benchmarks," the state bidding documents say. "And will, among other things, serve as the trigger mechanism for initiating payments and assessing liquidated damages, if applicable."
For most of the testing firms considering the state contract, those rigid deadlines were too tough to accept.
"That was a real killer, I think, for a lot of people," said Dan Stone, president of Testcom Inc., one of the nine firms that considered but did not bid on the state's inspection contract. "Had we sensed there was some flexibility in schedules and a less punitive approach to penalties, yeah, it might have changed our perception regarding bidding or not bidding."
A spokesman for Keating Technologies said the state's hard stance on the proposed deadlines had discouraged the company from submitting a bid.
During pre-bid conferences, state records show, Keating unsuccessfully sought a three-month extension on the Oct. 1 deadline for the computer hookup. Keating also asked whether a firm would face the $33,334 per-day fine for failing to complete just one of the 36 inspection lanes required months before the Dec. 13 deadlines, and was told in writing, "Yes."
"I think, had the deadlines been different at that time, we might have considered bidding it a little more seriously," said Chris Stock, a spokesman for Keating, which operates testing programs in Rhode Island and Massachusetts. "Our principal concern was the time frames to do the job, and the other work we had."
Stanley Rosenblum, a former state transportation officer who helped develop the bid, known as a "request for proposal," denied that the deadlines scared away companies.
'I reject the concept that our RFP drove prospective bidders out of the system," he said earlier this month. "Those dates were being driven by the reality that we had to get a program in place." But a consultant hired to serve as watchdog over Parsons had warned the Treasury Department its bidding standards were too tough. "We have identified the following two major concerns that we believe may discourage the most qualified and responsible firms from submitting bids," Sierra Research Corp. of Sacramento, Calif., said in a memo written Nov. 12, 1997 -- four months before the RFP was published. Sierra cited uncertain estimates of inspection volume and "an unrealistic implementation schedule that severely jeopardizes a successful program start-up."
Sierra is a subcontractor of Parsons Brinckerhoff, a Princeton consulting company that has a state contract to oversee the testing system. Parsons Brinckerhoff and Parsons Infrastructure are unrelated companies.
The 1997 memo from Sierra was prophetic. Parsons Infrastructure was the only firm to bid for the state's inspection program. And it was far from a reluctant bidder.
New Jersey's massive car inspection contract, which accommodates more than 5 million vehicles and can generate more than $50 million in annual fees, has long been a coveted prize. Since 1992, firms that run such systems have sought a foothold in New Jersey. Between 1994 and 1996, state records show, three testing firms paid New Jersey lobbyists a total of $173,085. Envirotest, the nation's largest private inspection company, paid $116,650 to two key Trenton lobbyists: Hazel Gluck, who had co-chaired Whitman's 1993 gubernatorial campaign, and Dale Florio, chairman of the Somerset County Republican Committee.
When the state's first bid proposal drew little interest, Parsons Infrastructure entered the game. It entered late but it played hard. The Pasadena, Calif., company, a subsidiary of Parsons Corp., had engineered such massive projects as the Alaska Pipeline and the tunnel beneath the English Channel. It began laying the groundwork for a run at New Jersey's contract.
The firm hired Frank Holman, former chairman of the Republican State Committee, to keep it apprised of developments in the emissions contract.
Parsons Infrastructure recruited Carl Golden, a former spokesman for Whitman and another paid consultant to the Republican State Committee, to help develop its testing bid. And the firm retained Roger Bodman, a former state transportation commissioner and one of Trenton's top GOP lobbyists, to work on its behalf.
Parsons Infrastructure also poured $141,800 into New Jersey Republican Party campaign chests during 1997 and 1998, state records show.
Its bid for the inspection contract, delivered in Trenton on June 12, 1998, contained names familiar to most state officials.
Golden was named the program's "public information and education administrator," and the advertising firm where he worked was awarded a $15 million contract. Anthony Sartor, an engineer who is a personal and business associate of Senate President Donald DiFrancesco (R-Union), was assigned a $3 million engineering subcontract under the Parsons bid.
Lawmakers, including DiFrancesco, raised objections. They complained the Parsons bid was costly, and, with a minimum of $50 million a year in operating costs, far more expensive than the old state-run system. State union leaders, who opposed privatization of the central inspection stations, contended they could run the system for $11 million less than Parsons proposed, or a savings of $77 million over the life of the contract.
After a three-hour Senate hearing, at which then-state Treasurer James DiEleuterio defended the Parsons Infrastructure deal, the state signed the seven-year pact on Aug. 7, 1998.
"By any measurement, Parsons' qualifications and its experience are quite impressive," DiEleuterio said.
Golden scoffs at any suggestion that he was hired to help Parsons Infrastructure get an inside track on the contract. Neither Sartor or Bodman has returned phone calls, and Holman says he played little role in Parsons getting the contract.
The seven-year deal with Parsons was originally projected to cost a total of $392 million, state contract records show: $63 million for overhauling the inspection system and installing new equipment and $329 million to run it for seven years.
During its first year under the contract, however, the firm has received $103 million. That total includes $43 million for construction and equipment costs and about $60 million in operating expenses.
The operating costs far exceed the $34 million the state estimated for the first year, when Parsons was conducting the state's older tailpipe emissions test. The state originally projected that Parsons would be testing 1.7 million cars in the first year. But that projection proved to be low by 600,000 cars.
Using those inspection totals projected over the next six years, plus the expectation that the numbers of cars in the state will increase each year, the entire cost of the contract may well approach $592 million.
It was not long after Parsons won the contract that it was allowed to miss the deadlines that were so emphatically stated in the state's specifications. Within a month of the deal, the state scrapped its Oct. 1 deadline, records show.
Parsons missed another critical deadline on July 11, 1999, when it failed to get its testing lanes ready to operate. The state did not penalize Parsons as it had warned potential bidders it would. Carl Passeri, the state Division of Motor Vehicles official in charge of the inspection contract, last year explained that the interim deadlines had been just "a goal," and said the Oct. 1 deadline was "an unrealistic date that was put out there in haste." Passeri has declined to comment on any Parsons matter for the past several weeks.
Parsons Infrastructure says the deadlines were postponed because the Department of Environmental Protection and other state agencies were late in defining what the system's computer programs needed to include.
Sierra Research, the watchdog consultant, noted that Parsons Infrastructure was failing to get required work done on time, and suggested that if the state continued to let deadlines slip, New Jersey would either miss the federal deadline or would force an incomplete, inadequate system on motorists.
To Sierra, the state was allowing Parsons to push it around. "It is also apparent that the state is being tested by PI&TG (Parsons Infrastructure), and the state is failing the test," Sierra warned in a Dec. 29, 1998, memo.
Parsons "is determining the extent to which it needs to be responsive to the state, and thus far the company is seeing that failure to follow through on commitments or resolve technical issues results in no adverse consequences."
That memo from Sierra, hired by consultant Parsons Brinckerhoff, was one of nearly 100 that the research company wrote over two years about the testing program's progress, or lack of it. The existence of the memos became public last week when The Star-Ledger asked to see them and the Governor declared she had become aware of them for the first time.
Golden said Parsons Infrastructure never saw the memos.
"This is an issue between Sierra and Parsons Brinckerhoff," said Golden. Parsons Infrastructure "was not privy to those communications."
Larry Sherwood, Parsons Infrastructure general manager in New Jersey, said Friday night of the memos, "I have yet to see them." He declined further comment, saying it was a Sierra and Parsons Brinckerhoff matter.
A spokesman for Parsons Brinckerhoff said executives there "carried out all of their responsibilities under the oversight contract." He said summary reports on how the project was faring were sent to Passeri on a regular basis.
The memos, released by Whitman at her Thursday news conference, paint a portrait of an government whose oversight functions were on the blink.
"I suspect there is plenty of culpability to go around on this one," the Governor said during the news conference, at which she announced Attorney General John Farmer Jr. would conduct an inquiry into why the warnings in the memos never reached her.
Even after Thursday's bombshell, the contents of the memos continued to embarrass her administration. Whitman and Transportation Commissioner James Weinstein initially said it appeared that no members of the administration had seen the memos.
But a Star-Ledger review of the documents released Thursday shows at least 10 officials in the state departments of Environmental Protection or Transportation -- ranging from Passeri, the DMV official in charge of the inspection contract, to David West, a high-ranking DEP official -- received memos, e-mail messages or reports from Sierra Research. Passeri received at least three memos. On Friday, the administration acknowledged that some of the memos were sent to the bureaucrats in charge of making sure the new inspection system debuted smoothly.
Weinstein, who said Thursday that he did not believe that Passeri or anyone in state government had received the Sierra memos, reversed himself. Passeri did receive a small number of Sierra memos, he said. "There were some Sierra documents in his files," Weinstein said. Weinstein has accepted ultimate responsibility for the mess. His lieutenant, DMV Director Richard Kamin, also said he had not read any Sierra memos until this weekend. He added that he and others were aware of the general areas of concern, in part because of DMV's own experience running a test inspection lane in Wayne and in part because software glitches and other problems had shown up at the start of other states' inspection programs.
"We were confident we could make it work" because dozens of people were laboring on the program, Kamin said.
Kamin declined to go into other aspects of the process, citing the pending attorney general's probe.
Another problem signaled early, and subsequently ignored, was manpower.
In a 1997 forecast of inspection needs, the state DMV said running a testing system with 106 lanes would require 957 employees. Parsons, by contract, proposed to operate 124 testing lanes with 530 workers. A chronic shortage of inspectors has compounded the problems for motorists. For most of 1999, when Parsons was paid about $54 million for administering the state's traditional inspection test, the firm had fewer than 400 workers in the testing centers.
When the new, more complicated, test debuted, the firm had 548 inspectors on hand, according to Golden. That is about the number promised in Parsons' bid for the work. Now Parsons is trying to hire 442 additional workers.
The additions will bring the staff to the level state experts long ago projected as needed to run an enhanced testing program. But the apparent miscalculation has meant continued short-staffing while Parsons spends weeks recruiting, training and certifying hundreds of new inspectors.
"It was always recognized by Parsons that if they did not have enough staffing they were responsible," Rosenblum said.
Adding to insult for motorists was the absence of sheltered waiting areas. With drivers spending upwards of 20 minutes outside their cars during the test, Whitman demanded to know what had happened to the sheltered, heated waiting rooms required by state law.
In fact, Treasury Department officials say, the requirement for the waiting rooms had been written out of the bidder's specifications, as part of a series of changes designed to keep the cost of the new inspection program down after the first failed bid effort in 1997. Instead, the bid required contractors to "make those repairs and modifications to the inspection facilities that are necessary to make them comfortable and convenient to customers." Under fire from Whitman and the public, Parsons Infrastructure this month began installing bus-stop-like shelters at the end of each inspection lane. There was more.
Computerized scanners designed to read coded information about a car's engine size, weight, age and other features from codes on state registration cards didn't work. Technicians resorted to punching in information by hand. But vehicle identification numbers are 17 digits long. Many of the VINs had to be punched and repunched because of manual errors.
On frequent occasions, once the car was identified, the computerized inspection equipment lost the information. The computer that was supposed to read a car's make or weight and transfer it to stations farther down the testing lane was sending incorrect numbers. As a result, some cars were flunking brake tests they should have passed. Tires blew out on cars being tested on the dynamometer.
When temperatures dipped into the teens this month, Parsons Infrastructure found that much of the sensitive computer software and emissions testing devices installed in its testing lanes froze up. State records show the contract required that equipment be tested to a low temperature of only 40 degrees. Sherwood said Parsons Infrastructure was not concerned about cold weather disabling its machinery.
"We were told that the system was identical to Connecticut, so we really didn't worry about the freezing and temperature impacts," he said.
A memo from Sierra Research last January warned of potential problems in temperatures below 35 degrees.
Last week, Whitman conceded that warnings had gone unheeded.
"We're looking at a total system failure," Whitman said Thursday. "If I had thought for one second (such problems would be coming), I would have gone back to the EPA saying, 'You've got to give us more time.'" Parsons Infrastructure has no plans to ask the state to delay the program, Golden said, adding it is the administration's call on whether to halt it.
"We will do whatever are the rules and regulations imposed on us," he said.
Staff writers Brian Murray, Tom Johnson and Brian Donohue contributed to this report.
Link to story:
-- Carl Jenkins (Somewherepress@aol.com), January 24, 2000
Carl- You are tb2000's busiest researcher-that is appreciated. But I do have a problem with your headline on this one. It is unnecessarily alarmist and gave the impression that it was somehow y2k related-And I know you use OT often enough-this time you didn't-this is just a typical bureacratic snafu that was occuring before computers were used-I live in NJ so woe to me-but this is definately not on topic.
-- futureshock (email@example.com), January 24, 2000.
Hi futureshock. I don't think the headline is overstated at all. The Governor of New Jersey said:
Don't know if y2k issues are specifically involved, but some parts of long complicated report sound suspiciously like the sorts of things anticipated from y2k:
"We're looking at a total system failure," Whitman said Thursday"
"On frequent occasions, once the car was identified, the computerized inspection equipment lost the information. The computer that was supposed to read a car's make or weight and transfer it to stations farther down the testing lane was sending incorrect numbers. As a result, some cars were flunking brake tests they should have passed."
and so on...
I stand by the post as a possible y2k related incident in need of additional investigation.
-- Carl Jenkins (Somewherepress@aol.com), January 24, 2000.
Sorry...I garbled first part of reponse...
I don't think the headline is overstated at all. The Governor of New Jersey said: "We're looking at a total system failure," Whitman said Thursday"
If anything, the headline is buried at the end of this long and confusiong story.
-- Carl Jenkins (Somewherepress@aol.com), January 24, 2000.
I understand better where you are coming from. I interpreted the word "system" to cover the whole process, from bidding to the current problem, and not just the computer. Inability to send the correct information down the line could be any number of things, but I do not think it would be the first thing to look at; I guess since y2k has not been ruled out, it remains a possibility, however remote, but I would suspect it is not-I just think there was just bad programming from the start-Only my opinion. But I do appreciate your work. Peace
-- futureshock (firstname.lastname@example.org), January 24, 2000.
They also have a severe understaffing problem. Just saying... NOW they want you to make an appointment for your inspection. So people who just drive up have to wait in even LONGER lines. Enormous sheesh!
-- kirsten (email@example.com), January 24, 2000.
I'm guessing that there have been similar breakdowns in other government computer systems but we haven't heard about them because they don't interact so much with the public. I completely understand the anonymity issues at stake for whistleblowing, so therefore I ask that we allow a brief episode of unsubstantiated, anonymous, anecdotal posts about anyone who has experienced large breakdowns in their government (or private) jobs. I can't believe this New Jersey thing and the few others we've heard about is the extent of it. Remember, flames can't hurt you.
-- Flameaway (flameaway@idon'tcare.com), January 24, 2000.
CARL and FUTURESHOCK:
Having been an extreme Doomer ("11.5" on a 0 to 10 scale) I should be the last one to question a link with this mess and Y2K. However I do.
Much more than having the earmarks of a Y2K disaster it presents the typical profile of a long litany of computer project failures down through the decades, from the '60s into the '90s, witnessed by so many of us who handled a soldering iron and/or a coding sheet (or if we were more fortunate, watched as our brethren "next door" were caught up in such fiascos.) There is no responsible worker in the field who would deny the ugly ubiquity of such catastrophes, LONG BEFORE 1/1/Oh!Oh! BTW, 'worker' does NOT include middle or upper management or politicians, or media commentators. ...
No, the verdict here will most likely be accurately handed down as "not Y2k-related." Thus I agree, starting the thread in this forum, at this time, with that headline, is misleading. But could there be any correlation other than the 'usual' between this big bust and the Y2K (so-far) mostly 'non-event?' Well, how about a negative correlation? --- how's that for Doomer heresy?
I'm gradually developing a sneaking suspicion that many of us Doomers mistook what would be involved with much of the Y2K remediation ... with the standard metrics typical of so much of the industry, based on designing computer systems de novo, totally from scratch: grandiose management projections coupled with institutionalized stonewalling of danger signals from the trenches, chronically slipped deadlines, and topped off with the inevitable budget overruns.
I admit I would occasionally read posts making the above distinction -- but I ignored them. Why? Mistrust of the optimists? A bias towards Doom? (Maybe I'll write to that -- after seeing whether EOM, EOQ brings up any submerged icebergs. If no icebergs appear SOME one of us Doomers needs to address that issue.)
Bill -- a more contemplative than previous, Doomer.
-- William J. Schenker, MD (firstname.lastname@example.org), January 24, 2000.