เมื่อเร็วๆ นี้ (วันที่ 5 กันยายน 2552) เพื่อนผมท่านหนึ่งได้ส่งข่าวมาให้อ่าน “ตลาดหุ้นเสีย วอลุ่ม ทันที 4 พันล้านบาท ” ในเนื้อข่าวไม่ได้บอกอะไรมากในแง่ของข้อมูล Technical แต่เพื่อนผมที่เคยทำงานอยู่ ณ ที่บริษัทที่ทำ Software ซื้อขายหุ้นตัวทีมีปัญหาตามข่าว บอกว่าเป็นปัญหาเรื่องของระบบ และการทดสอบที่ไม่ครบถ้วน หรือ ถ้าเพื่อนพ้องน้องพี่ยังจำข่าวเมื่อราวๆ ปีกว่าๆ ได้ที่มี ชาวนาที่อยุธยา เติมเงินเข้ามือถือ แต่พอเช็คยอดแล้วมีเงินในมือถือตัวเองหลักล้านบาท ทั้ง 2 กรณีที่ผู้เขียนยกขึ้นมานั้นถ้ามองในแง่ของธุรกิจ เกิดความเสียหายขึ้นทันที ผลที่เกิดขึ้นนั้นมาจาก Bug ที่เกิดขึ้นใน Software

ตลอดระยะเวลาเกือบ 5 ปี ที่ผู้เขียนทำงานเกี่ยวกับ Software Testing ก็พบเหตุการณ์ที่ก่อให้เกิดความเสียหายกับธุรกิจ อันเนื่องมาจาก Bug ของ Software ด้วยเช่นกัน ไม่ว่าจะประสบมากับตัวเอง หรือจากการเล่าสู่กันฟังของเพื่อนพ้องน้องพี่ที่ทำงานในสายงานเดียวกัน แต่ทั้งนี้ทั้งนั้นก็มิได้จะหมายความว่าเราจะชี้นิ้วด่าลงไปที่ Programmer หรือ Developer หรือแม้แต่ Tester ว่าทำงานไม่ดี ทำงานไม่ได้เรื่อง เสียทีเดียว มันมีหลายๆ ปัจจัย ที่เกี่ยวข้อง มีคนเคยบอกไว้ว่า “Quality is Expensive” ทำให้หลายๆ องค์กร มองข้ามเรื่องนี้ไป หรือไม่ค่อยให้ความสำคัญสักเท่าไรนัก แต่พอเกิดปัญหาขึ้น ก็พยายามหาที่ Landing หรือ แพะ ทันที ซึ่งส่วนมากก็จะมาลงที่ทีม Development หรือไม่ก็ทีม Test (ความเห็นส่วนตัวครับ)

ผู้เขียนจำได้ว่าเคยอ่านข้อมูลมาจาก Website หนึ่ง สมัยเริ่มทำงานด้าน Software Testing ใหม่ๆ คลับคล้ายคลับคลา ว่ามีหัวข้อที่ยกตัวอย่างผลกระทบที่เกิดขึ้นกับธุรกิจอันเนื่องมาจาก Bug ของ Software ดังนั้นจึงลองไปค้นๆ เอกสารดู และก็เจอจนได้ ข้อมูลอ่านจะเก่าไปสักหน่อย เพราะวันที่ที่ผู้เขียนสั่งพิมพ์ไว้ ลง วันที่ 27/3/2549 16:47 บทความเขียนขึ้นเมื่อไรนี่ไม่รู้นะครับ เดชะบุญที่ยังมี URL ของบทความอยู่บนกระดาษ ผู้เขียนเลยตามเข้าไปดู URL ยังสามารถเข้าได้ และมีข้อมูล Update ล่าสุดเมื่อ February 23, 2009 ซึ่งต้นฉบับเป็นข้อมูลภาษาอังกฤษ ดังนั้นผู้เขียนจึงขออนุญาตินำเสนอเป็นข้อมูลภาษาอังกฤษด้วยเช่นกัน

What are some recent major computer system failures caused by software bugs?

Reference: Software QA and Testing Resource Center – FAQ Part 1

• In February of 2009 users of a major search engine site were prevented
  from clicking through to sites listed in search results for part of a day.
  It was reportedly due to software that did not effectively handle a mistakenly-placed
  “/” in an internal ancillary reference file that was frequently updated for use by the
  search engine. Users, instead of being able to click thru to listed sites, were
  instead redirected to an intermediary site which, as a result of the suddenly
  enormous load, was rendered unusable.
• A large health insurance company was reportedly banned by regulators from
  selling certain types of insurance policies in January of 2009 due to ongoing computer
  system problems that resulted in denial of coverage for needed medications
  and mistaken overcharging or cancelation of benefits. The regulatory agency was
  quoted as stating that the problems were posing “a serious threat to the health
  and safety” of beneficiaries.
• A news report in January 2009 indicated that a major IT and management
  consulting company was still battling years of problems in implementing its own
  internal accounting systems, including a 2005 implementation that
  reportedly “was attempted without adequate testing”.
• In August of 2008 it was reported that more than 600 U.S. airline flights
  were significantly delayed due to a software glitch in the U.S. FAA air traffic
  control system. The problem was claimed to be a ‘packet switch’ that ‘failed
  due to a database mismatch’, and occurred in the part of the system that handles
  required flight plans.
• Software system problems at a large health insurance company in August 2008 were
  the cause of a privacy breach of personal health information for several hundred thousand
  customers, according to news reports. It was claimed that the problem was
  due to software that ‘was not comprehensively tested’.
• A major clothing retailer was reportedly hit with significant software and
  system problems when attempting to upgrade their online retailing systems
  in June 2008. Problems remained ongoing for some time. When the company made
  their public quarterly financial report, the software and system problems were
  claimed as the cause of the poor financial results.
• Software problems in the automated baggage sorting system of a major airport
  in February 2008 prevented thousands of passengers from checking baggage
  for their flights. It was reported that the breakdown occurred during a
  software upgrade, despite pre-testing of the software. The system continued
  to have problems in subsequent months.
• News reports in December of 2007 indicated that significant software
  problems were continuing to occur in a new ERP payroll system for a
  large urban school system. It was believed that more than one third of
  employees had received incorrect paychecks at various times since the
  new system went live the preceding January, resulting in overpayments
  of $53 million, as well as underpayments. An employees’ union brought
  a lawsuit against the school system, the cost of the ERP system was
  expected to rise by 40%, and the non-payroll part of the ERP system was
  delayed. Inadequate testing reportedly contributed to the problems.
• In November of 2007 a regional government reportedly brought a
  multi-million dollar lawsuit against a software services vendor,
  claiming that the vendor ‘minimized quality’ in delivering
  software for a large criminal justice information system and the
  system did not meet requirements. The vendor also sued its
  subcontractor on the project.
• In June of 2007 news reports claimed that software flaws in a popular
  online stock-picking contest could be used to gain an unfair advantage
  in pursuit of the game’s large cash prizes. Outside investigators
  were called in and in July the contest winner was announced. Reportedly
  the winner had previously been in 6th place, indicating that the
  top 5 contestants may have been disqualified.
• A software problem contributed to a rail car fire in a major underground metro
  system in April of 2007 according to newspaper accounts. The software
  reportedly failed to perform as expected in detecting and preventing excess
  power usage in equipment on new passenger rail cars, resulting in overheating and
  fire in the rail car, and evacuation and shutdown of part of the system.
• Tens of thousands of medical devices were recalled in March of 2007 to
  correct a software bug. According to news reports, the software would not
  reliably indicate when available power to the device was too low.
• A September 2006 news report indicated problems with software
  utilized in a state government’s primary election, resulting in
  periodic unexpected rebooting of voter checkin machines, which
  were separate from the electronic voting machines, and resulted
  in confusion and delays at voting sites. The problem was reportedly
  due to insufficient testing.
• In August of 2006 a U.S. government student loan service
  erroneously made public the personal data of as many as 21,000 borrowers
  on it’s web site, due to a software error. The bug was fixed and the
  government department subsequently offered to arrange for free
  credit monitoring services for those affected.
• A software error reportedly resulted in overbilling of up to several
  thousand dollars to each of 11,000 customers of a major telecommunications
  company in June of 2006. It was reported that the software bug was fixed
  within days, but that correcting the billing errors would take much longer.
• News reports in May of 2006 described a multi-million dollar lawsuit
  settlement paid by a healthcare software vendor to one of its customers.
  It was reported that the customer claimed there were problems with the
  software they had contracted for, including poor integration of software
  modules, and problems that resulted in missing or incorrect data used by
  medical personnel.
• In early 2006 problems in a government’s financial monitoring software
  resulted in incorrect election candidate financial reports being made
  available to the public. The government’s election finance
  reporting web site had to be shut down until the software was repaired.
• Trading on a major Asian stock exchange was brought to a halt
  in November of 2005, reportedly due to an error in a system
  software upgrade. The problem was rectified and
  trading resumed later the same day.
• A May 2005 newspaper article reported that a major hybrid car
  manufacturer had to install a software fix on 20,000 vehicles
  due to problems with invalid engine warning lights and
  occasional stalling. In the article, an automotive software
  specialist indicated that the automobile industry spends $2 billion
  to $3 billion per year fixing software problems.
• Media reports in January of 2005 detailed severe problems with
  a $170 million high-profile U.S. government IT systems project. Software
  testing was one of the five major problem areas according to a
  report of the commission reviewing the project. In March of 2005
  it was decided to scrap the entire project.
• In July 2004 newspapers reported that a new government
  welfare management system in Canada costing several hundred million
  dollars was unable to handle a simple benefits rate increase after
  being put into live operation. Reportedly the original contract
  allowed for only 6 weeks of acceptance testing and the system was
  never tested for its ability to handle a rate increase.
• Millions of bank accounts were impacted by errors due to installation
  of inadequately tested software code in the transaction processing
  system of a major North American bank, according to mid-2004 news
  reports. Articles about the incident stated that it took two weeks
  to fix all the resulting errors, that additional problems resulted
  when the incident drew a large number of e-mail phishing attacks
  against the bank’s customers, and that the total cost of the incident
  could exceed $100 million.
• A bug in site management software utilized by companies
  with a significant percentage of worldwide web traffic was
  reported in May of 2004. The bug resulted in performance
  problems for many of the sites simultaneously and required
  disabling of the software until the bug was fixed.
• According to news reports in April of 2004, a software bug was
  determined to be a major contributor to the 2003 Northeast
  blackout, the worst power system failure in North American
  history. The failure involved loss of electrical power to
  50 million customers, forced shutdown of 100 power plants,
  and economic losses estimated at $6 billion. The bug was
  reportedly in one utility company’s vendor-supplied power
  monitoring and management system, which was unable to correctly
  handle and report on an unusual confluence of initially localized
  events. The error was found and corrected after examining
  millions of lines of code.
• In early 2004, news reports revealed the intentional use
  of a software bug as a counter-espionage tool. According to the
  report, in the early 1980′s one nation surreptitiously allowed a hostile
  nation’s espionage service to steal a version of sophisticated
  industrial software that had intentionally-added flaws. This
  eventually resulted in major industrial disruption in the country
  that used the stolen flawed software.
• A major U.S. retailer was reportedly hit with a large government fine
  in October of 2003 due to web site errors that enabled customers to
  view one anothers’ online orders.
• News stories in the fall of 2003 stated that a manufacturing company
  recalled all their transportation products in order to fix a software
  problem causing instability in certain circumstances. The company found
  and reported the bug itself and initiated the recall procedure in which
  a software upgrade fixed the problems.
• In August of 2003 a U.S. court ruled that a lawsuit against a large
  online brokerage company could proceed; the lawsuit reportedly
  involved claims that the company was not fixing system problems
  that sometimes resulted in failed stock trades, based on the
  experiences of 4 plaintiffs during an 8-month period. A previous
  lower court’s ruling that “…six miscues out of more than
  400 trades does not indicate negligence.” was invalidated.
• In April of 2003 it was announced that a large student loan company
  in the U.S. made a software error in calculating the monthly
  payments on 800,000 loans. Although borrowers were to be notified
  of an increase in their required payments, the company will still
  reportedly lose $8 million in interest. The error was uncovered
  when borrowers began reporting inconsistencies in their bills.
• News reports in February of 2003 revealed that the U.S. Treasury
  Department mailed 50,000 Social Security checks without any beneficiary
  names. A spokesperson indicated that the missing names were due
  to an error in a software change. Replacement checks were
  subsequently mailed out with the problem corrected, and recipients
  were then able to cash their Social Security checks.
• In March of 2002 it was reported that software bugs in Britain’s
  national tax system resulted in more than 100,000 erroneous tax
  overcharges. The problem was partly attributed to the difficulty of
  testing the integration of multiple systems.
• A newspaper columnist reported in July 2001 that a serious flaw was
  found in off-the-shelf software that had long been used in systems
  for tracking certain U.S. nuclear materials. The same software had been
  recently donated to another country to be used in tracking their own
  nuclear materials, and it was not until scientists in that country
  discovered the problem, and shared the information, that U.S.
  officials became aware of the problems.
• According to newspaper stories in mid-2001, a major systems
  development contractor was fired and sued over problems with a
  large retirement plan management system. According to the reports,
  the client claimed that system deliveries were late, the software had
  excessive defects, and it caused other systems to crash.
• In January of 2001 newspapers reported that a major European
  railroad was hit by the aftereffects of the Y2K bug. The company
  found that many of their newer trains would not run due to their
  inability to recognize the date ’31/12/2000′; the trains were
  started by altering the control system’s date settings.
• News reports in September of 2000 told of a software vendor
  settling a lawsuit with a large mortgage lender; the vendor had
  reportedly delivered an online mortgage processing system that
  did not meet specifications, was delivered late, and didn’t work.
• In early 2000, major problems were reported with a new computer
  system in a large suburban U.S. public school district with 100,000+
  students; problems included 10,000 erroneous report cards and students
  left stranded by failed class registration systems; the district’s
  CIO was fired. The school district decided to reinstate it’s original
  25-year old system for at least a year until the bugs were worked out
  of the new system by the software vendors.
• A review board concluded that the NASA Mars Polar Lander failed in
  December 1999 due to software problems that caused improper functioning
  of retro rockets utilized by the Lander as it entered the Martian atmosphere.
• In October of 1999 the $125 million NASA Mars Climate
  Orbiter spacecraft was believed to be lost in space due
  to a simple data conversion error. It was determined that
  spacecraft software used certain data in English units that should
  have been in metric units. Among other tasks, the orbiter
  was to serve as a communications relay for the Mars
  Polar Lander mission, which failed for unknown reasons
  in December 1999. Several investigating panels were
  convened to determine the process failures that allowed
  the error to go undetected.
• Bugs in software supporting a large commercial high-speed data
  network affected 70,000 business customers over a period of 8 days
  in August of 1999. Among those affected was the electronic trading
  system of the largest U.S. futures exchange, which was shut down
  for most of a week as a result of the outages.
• In April of 1999 a software bug caused the failure of a $1.2 billion
  U.S. military satellite launch, the costliest unmanned accident in the
  history of Cape Canaveral launches. The failure was the latest
  in a string of launch failures, triggering a complete military
  and industry review of U.S. space launch programs, including software
  integration and testing processes. Congressional oversight hearings
  were requested.
• A small town in Illinois in the U.S. received an unusually large monthly
  electric bill of $7 million in March of 1999. This was about 700
  times larger than its normal bill. It turned out to be due to
  bugs in new software that had been purchased by the local power
  company to deal with Y2K software issues.
• In early 1999 a major computer game company recalled all copies
  of a popular new product due to software problems. The company
  made a public apology for releasing a product before it was ready.
• The computer system of a major online U.S. stock trading service
  failed during trading hours several times over a period of days in
  February of 1999 according to nationwide news reports. The problem
  was reportedly due to bugs in a software upgrade intended to
  speed online trade confirmations.
• In April of 1998 a major U.S. data communications network
  failed for 24 hours, crippling a large part of some U.S. credit
  card transaction authorization systems as well as other large U.S.
  bank, retail, and government data systems. The cause was
  eventually traced to a software bug.
• January 1998 news reports told of software problems at a
  major U.S. telecommunications company that resulted in no charges
  for long distance calls for a month for 400,000 customers. The
  problem went undetected until customers called up with
  questions about their bills.
• In November of 1997 the stock of a major health industry
  company dropped 60% due to reports of failures in computer
  billing systems, problems with a large database conversion,
  and inadequate software testing. It was reported that more than
  $100,000,000 in receivables had to be written off and that
  multi-million dollar fines were levied on the company by
  government agencies.
• A retail store chain filed suit in August of 1997
  against a transaction processing system vendor (not a credit
  card company) due to the software’s inability to handle
  credit cards with year 2000 expiration dates.
• In August of 1997 one of the leading consumer credit reporting
  companies reportedly shut down their new public web site after
  less than two days of operation due to software problems. The new
  site allowed web site visitors instant access, for a small
  fee, to their personal credit reports. However, a number of
  initial users ended up viewing each others’ reports instead
  of their own, resulting in irate customers and nationwide
  publicity. The problem was attributed to “…unexpectedly
  high demand from consumers and faulty software that routed
  the files to the wrong computers.”
• In November of 1996, newspapers reported that software bugs caused
  the 411 telephone information system of one of the U.S. RBOC’s to
  fail for most of a day. Most of the 2000 operators had to
  search through phone books instead of using their 13,000,000-listing
  database. The bugs were introduced by new software modifications
  and the problem software had been installed on both the production
  and backup systems. A spokesman for the software vendor reportedly
  stated that ‘It had nothing to do with the integrity of the
  software. It was human error.’
• On June 4 1996 the first flight of the
  European Space Agency’s new Ariane 5 rocket failed shortly
  after launching, resulting in an estimated uninsured loss
  of a half billion dollars. It was reportedly due to the lack
  of exception handling of a floating-point error in a
  conversion from a 64-bit integer to a 16-bit signed integer.
• Software bugs caused the bank accounts of 823 customers of a major
  U.S. bank to be credited with $924,844,208.32 each in May of 1996,
  according to newspaper reports. The American Bankers Association
  claimed it was the largest such error in banking history. A bank
  spokesman said the programming errors were corrected and all
  funds were recovered.
• In August 1991 the concrete base structure for a North Sea oil platform
  imploded and sank off the coast of Norway, reportedly due to errors in
  initially-used design software. The enormous structure, on hitting the seabed,
  reportedly was detected as a magnitude 3.0 seismic event and resulted in a loss
  of $700 million. The base structure was eventually redesigned and the full
  platform was completed two years later, and was still in use as of 2008.
• On January 1 1984 all computers produced by one of the
  leading minicomputer makers of the time reportedly failed worldwide.
  The cause was claimed to be a leap year bug in a date handling function
  utilized in deletion of temporary operating system files. Technicians
  throughout the world worked for several days to clear up the problem.
  It was also reported that the same bug affected many of the same
  computers four years later.
• Software bugs in a Soviet early-warning monitoring system
  nearly brought on nuclear war in 1983, according to news reports
  in early 1999. The software was supposed to filter out
  false missile detections caused by Soviet satellites picking up
  sunlight reflections off cloud-tops, but failed to do so. Disaster was
  averted when a Soviet commander, based on what he said was a ‘…funny
  feeling in my gut’, decided the apparent missile attack was a
  false alarm. The filtering software code was rewritten.

Quality is Expensive, Are you dare to have risk on your business?