A flop; a failure; a disaster. They happen and even when the warning bells sound, the project team forges ahead. US Computerworld reporter Kim Nash explores the American experienceComputer projects have failed for as long as there have been computers. But now that most companies are only as stable as their bits and bytes, the consequences of information technology screwups aren't easily disguised - they show up in earnings reports.
When IT goes bad, high-growth rocket ships report their first-ever financial losses. Others crater and run for bankruptcy protection, as did drug distribution giant FoxMeyer Corp.
In a US Computerworld study of multimillion-dollar IT disasters, the following two not-so-surprising themes emerged:
* User companies like FoxMeyer often file tough-to-win lawsuits against the vendors or consultants involved. Nonetheless, collectively, users rarely seem to learn much from the episodes or apply the lessons to future projects.
* All of the botched projects in US Computerworld's disasters list were big and richly complex; many were the toughest IT projects the users had ever tried. Five were hideously difficult enterprise resources planning (ERP) system implementations.
Root causes remain the same
The root causes of IT failures haven't changed a bit over the years.
Miscommunication, hazy goals, "scope creep", inept leadership, pitiful project management - you've heard, if not heeded, it all before.
"We may be neck-deep in the New Economy and Internet time, but you still have the same factors and the same failings," said Bruce Webster, a director at PricewaterhouseCoopers.
Webster recently studied 120 IT lawsuits filed in the US since 1976, and he said he's convinced that most flops could be avoided if people simply knew the time-honoured best practices of systems development.
"I don't know how many IT managers, team leaders, directors and CIOs have actually sat down and read The Mythical Man-Month, The Psychology of Computer Programming and Death March," he said, referring to three books that amount to the software development canon. "The causes of disasters are all well documented. They're fundamental."
Still, warning lights are easy to overlook when the whole room is spinning.
"There's a natural tendency to get overly committed to something, especially when there are no clear signals telling you you are off course," said Mark Keil, an associate professor at Georgia State University in Atlanta.
The infamously buggy baggage-handling system at the Denver International Airport is one case that offered unambiguous proof of technology glitches: shredded luggage.
But tests of most questionable IT projects don't yield such graphic evidence.
Project euthanasia
In large systems integration or ERP deals, "there's no torn suitcase sitting at your feet to wake you up", said Keil, who has studied IT disasters for nine years. "So it's a lot easier to delude yourself into thinking things aren't that bad."
Euthanasia for the project might be the best course, but people often have too much heart and money invested to end it.
One technique for preventing a disaster is to add some humility to the endeavour: invite a third party to review your work - a reliable consultant, an academic or a buddy CIO.
An outsider can "walk into the project setting for 20 minutes, talk to a few people and come to the conclusion that things have run amok. But people inside may not even be aware," Keil said.
Greyhound Lines in the US, for example, seemingly didn't know anything was wrong with its new reservations and logistics system - until it went live and 12 per cent of its customers went away angry in one month.
Though specific individuals might learn from their own mistakes, those lessons aren't transferred to any collective IT consciousness.
"The people with that [failure] experience aren't always the people in authority the next time that situation arises," observed Kevin Hickey, a former head of IT at Oxford Health Plans. "The fact is, hubris will always be with us."
And then there's what Webster calls "the thermocline of truth". Swimmers know that lake water separates into warm and cold horizontal bands. The area between is a thermocline.
In IT groups, everyone below Webster's "thermocline of truth" knows the project is sinking, while everyone above it thinks things are fine. Senior executives can be oblivious. They aren't involved enough, they don't want to have to face a failure, or underlings are afraid to tell them, he explained.
"You can see that persist almost until the point where the project is supposed to be delivered," he said.
"Then, suddenly, it's, 'What do you mean this will take another six months?'"That was part of the sorry plight of AMR Corp's Confirm reservations system.
Overall, IT culture is such that problems, especially expensive ones (which hold the most valuable lessons), are hidden. Programmers write around buggy code rather than tear it apart. Managers revise project specifications to reflect what they did instead of what they should have done. Senior IT leaders neglect to tell their bosses the bad news.
Most companies are too embarrassed to analyse their failures, said Effy Oz, an associate professor of management and IT at Pennsylvania State University.
"People will say, 'There's no time, and we're not paid to have these discussions'," Oz said. "The CEO has to be a very confident person to say, 'These things happen. Let's learn from it.'"The average loss in the US of an abandoned project is $4.2 million, according to Oz. The blowups in US Computerworld's list cost much more than that. And, if history is any indication, there will be more.
Dishonourable mentions
Some information technology disasters of the past decade aren't included in full here because their financial costs weren't clear or weren't quite as high as those that made the list. Others were quasi governmental. But they were significant bungles nonetheless. Here's a sampling:
* High hopes for a high-tech airport in Denver were dashed in 1994 when a baggage-sorting system misplaced and damaged countless suitcases. The object-oriented system, which was built by BAE Automated Systems of Texas, ran on IBM's OS/2.
To fix it, primary sponsors Chicago-based United Air Lines and the city of Denver ended up paying at least $US86 million more than the original $US193 million price tag. The airport opened almost three years late.
* Ben & Jerry's Homemade, a small Vermont ice cream maker that hit the big time in the early 1990s, took a $US4.1 million write-off in 1995 when it cancelled a project to build a fully automated packing plant. As a result, the company posted its first quarterly loss ever.
* Early this year, Thomas & Betts, a $US2.5 billion electrical parts maker in Memphis, blamed problems with a new Internet-based order-management system for a 50 per cent drop in profits in the fourth quarter of last year.
The homegrown mainframe system also cost the company another $US42 million in order and shipping disruptions. In the following quarter, the company spent $11 million on customer service, extra freight costs and other measures to make up for ongoing system crashes and backlogs.
In a lawsuit still pending, shareholders say Thomas & Betts misled them about the success of the new system.
* Just in time for a holiday crush in 1998, Avis' Wizcom reservation system crashed. The outage blocked the rental car company, as well as many travel agencies and hotels also linked to the system, from taking bookings for 30 hours.
* In the summer of 1994, an error in a routine file update crashed the automated teller machine (ATM) system of Chemical Banking Corp in New York. ATMs at Chemical, which was then one of the five largest banks in the US, were down for five hours.
* In 1993, Fidelity Brokerage Services in Boston started a multimillion-dollar project to build a Windows-based trading application for customers with home PCs. "Jamaica" still wasn't ready by mid-1996, reportedly because of political clashes between quirky programmers and staid investment bankers.
The application wasn't a failure; it worked when it was finally rolled out. But the numerous development delays put Fidelity far behind rivals such as Charles Schwab & Co. in San Francisco. - Kim NashTravel system is Confirm-ed disasterAMR's Confirm reservation system was a historic $US125 million flop On the heels of its hugely successful Sabre airline reservation system AMR Corp, the parent company of American Airlines, in the late 1980s formed a joint venture with Marriott International, Hilton Hotels and Budget Rent A Car to build a similar system for the travel industry. But Confirm, as the project was called, wasn't to be. In fact, the effort is viewed as one of the worst IT failures ever for its mismanagement, questionable ethics and unworkable software.
AMR's information systems unit in Dallas was the lead developer on Confirm, which was originally due in June 1992 for no more than $US55.7 million. Yet Confirm started to miss deadlines and cost estimates several months after development began.
Specifications were unclear, unskilled programmers were assigned to the project and mainframe-based transaction-processing software couldn't be integrated with a related mainframe decision-support system. Moreover, one year into design and development, Confirm had already fallen one year behind schedule.
Marriott and Lisle, and Budget started asking questions in 1990 but were assured that Confirm would work and that programmers would make up time and meet the deadline.
In April 1992, just three months before it was slated to go live, Confirm failed tests at Los Angeles-based Hilton. AMR also told its partners in a letter that it needed another 15 to 18 months.
"The individuals whom we gave responsibility for managing Confirm have proven to be inept [and] concealed a number of important technical and performance problems," the AMR letter said.
The legendary Max Hopper, who had been instrumental in Sabre's development, was vice chairman of AMR's IT unit at the time, though not directly involved in the daily work of Confirm. However, he acknowledged in a note to his employees that some Confirm managers "did not disclose the true status of the project in a timely manner. This has created more difficult problems - of both ethics and finance - than would have existed if those people had come forward with accurate information." After consuming almost four years and $125 million, Confirm was effectively dead.
In September 1992, AMR sued Budget, Hilton and Marriott; Marriott then sued AMR. The suits were settled out of court for undisclosed terms. Hopper recently declined to discuss Confirm, citing the secret settlements.
AMR was mainly to blame, but all sides acted unprofessionally, said Effy Oz, an associate professor of management and IT at Pennsylvania State University.
Budget, Hilton and Marriott today use their own - separate - reservation systems. - Kim NashA really bad bet for drug distributorBungled SAP project may have helped bankrupt FoxMeyerWhen it launched a $US35 million enterprise resource planning (ERP) project in 1993, FoxMeyer was a $5 billion drug distributor in Texas.
Now it's bankrupt.
It wasn't the fumbled IT project alone that destroyed FoxMeyer, but that was a critical contributor, according to lawsuits FoxMeyer filed against SAP AG, SAP America and Chicago-based Andersen Consulting in 1998.
The drug company claims SAP lied repeatedly about R/3's capabilities, and that Andersen's inexperienced staff used FoxMeyer as a training camp. The drug company seeks damages of $500 million from SAP and $500 million from Andersen.
FoxMeyer was one of the first big users to take a chance on ERP, which was a hot new idea at the time. Perhaps the company should have been more cautious. Legal documents show that FoxMeyer knew that SAP's R/3 software hadn't yet been used at distribution companies - just at manufacturers.
Still, FoxMeyer was jazzed. Then-CIO Robert Brown told US Computerworld in 1994, "We are betting our company on this."
Big problems started to emerge later that year. For example, the new R/3 software miscounted inventory, which in turn screwed up customer orders. Outright crashes were routine.
SAP declined to talk specifically about FoxMeyer. But an SAP spokesman said users who install R/3 are usually changing basic business processes at the same time, which "is typically where most of the pain and challenges of implementation come from".
FoxMeyer also charges that R/3 performed worse than the company's proprietary Unisys system. R/3 could process just 10,000 invoice lines per night, compared with 420,000 for the Unisys setup.
SAP misled FoxMeyer with faulty benchmarks, according to the suit. Other users have also questioned SAP's benchmarks. SAP doesn't misrepresent benchmarks, the company's spokesman said.
As for Andersen, according to FoxMeyer, many of its workers were recent college graduates, and others lacked R/3 experience.
They bungled data conversions by using incorrect drug product codes, for example, and built faulty interfaces between the old and new systems, the lawsuit charges.
As a result, most R/3 modules were rolled out to just six of 23 warehouses by the time the company filed for bankruptcy protection in August 1996.
Andersen didn't respond to requests for comments. The suits are slated for trials next May.
Meanwhile, FoxMeyer itself is being sued by stockholders, in part for allegedly hiding timely information about the computer problems and their effects on business. - Kim NashGrowers crop itTri Valley Growers says software scheme cost it $22 millionTri Valley Growers got caught in an Oracle software scheme that Oracle CEO Larry Ellison later admitted was a bad idea.
In November 1996, the $800 million agriculture cooperative in California bought $6 million worth of services and software from Oracle. The core of the deal was Oracle CPG, ERP software for consumer packaged goods companies.
While the financial and manufacturing pieces of the CPG suite were from Oracle, order processing, production planning and other packages were from other vendors. Oracle consultants were hired to integrate it all.
The new software was expected to make Tri Valley more efficient, improve customer service and save it $5 million a year.
In a press release trumpeting the Tri Valley deal in 1998, for example, Oracle said Tri Valley customers would be able to file a single purchase order, no matter how the cooperative managed its many fruit product lines internally.
But the system never got that far. Oracle couldn't get most of its applications to work with the non-Oracle software.
Tri Valley claims to have spent more than $22 million before halting the project and turning to SAP AG software.
The Oracle software "never worked, has not and cannot be integrated and could not even be installed on Tri Valley's computers", according to a lawsuit the cooperative filed in February.
Last year, Ellison called the Oracle CPG bundle "a huge mistake". Oracle no longer sells it.
In July, Tri Valley filed for bankruptcy protection. So far, it hasn't blamed its poor financial shape on the failed Oracle project - but it might, and it might ask for more money, to boot, said Peter Sipkins, Tri Valley's lawyer, with Dorsey & Whitney.
Oracle has denied all allegations. Tri Valley IT officials didn't return calls seeking comment. But a former IT manager there called the situation "very tragic".
A trial is scheduled for next June. - Kim NashHigh-Flying service modernises, crashesOxford's homegrown Pulse system led to red ink, unhappy customersOxford Health Plans was the Netscape of health maintenance organisations. It seemed to burst from nowhere, captivate customers and force competitors to change the way they operated.
Then Oxford decided to modernise its information technology.
It was 1995 and Oxford's old turnkey, Pick-based billing and membership tracking system would no longer do. A complete overhaul, using more modern Unix technology, is what the company wanted - and fast.
The project included many custom applications that ran with Oracle databases and other software. But key was a claims processing system, dubbed Pulse, that Oxford's internal IT people built with Oracle tools.
Trouble hit the company almost as soon as the rollout started in late 1996. Customers suddenly got claims laden with errors - when they got claims at all. The company paid bills it shouldn't have and denied claims it should have paid.
All the late and inaccurate paperwork caused New York state to fine Oxford $US3 million for violating insurance laws.
Overall, the new software overestimated revenue by $US392 million for 1997 and 1998 while also underestimating medical costs. That awful combination led to Oxford's $291 million loss in 1997.
Angry doctors and patients abandoned the company. Membership dropped 20 per cent from 1997 to 1999, partly because of the systems problems and partly because Oxford withdrew from four of the seven states it did business in.
Ultimately, top executives left, and Oxford hired a new head of operations - Kevin Hickey, then an operations manager at Aetna - to help with an IT cleanup already under way.
Hickey immediately faced down the billing mess, which "wasn't just an inconvenience. This was a survival issue", he said in an interview.
First, he shelved Pulse and returned to the old Pick application. Pulse was never fully integrated with the Oracle software, he said.
Cambridge Technology Partners and Diamond Technology Partners came in for quick-hit assignments to help fix claims processing.
Oxford also shut down its advanced technology unit. Studying artificial intelligence software for possible future systems was frivolous now that "emergency" IT problems threatened to incapacitate the company, Hickey explained.
Oxford hired Computer Sciences Corp to create a plan for outsourcing its entire IT operation.
But in 1998, a new CEO swept in and swept away that idea, along with most remaining legacy executives. Hickey, too, was replaced after just a year at the company.
Today, Oxford is smaller and smarter. It wrote off $5 million for hardware and software in 1998. Late last year, system fixes even took precedence over customer acquisition, according to documents filed with the US Securities and Exchange Commission (SEC).
Oxford has since completed major systems fixes and put in place new quality assurance and other testing programs. But SEC documents warn that unexpected sales calculations could still turn up. - Kim Nash