Audit of August Meltdown in Virginia Computer Systems Released

The McDonnell Administration and the Joint Legislative Audit and Review Committee (JLARC) have released the results of the external audit of the events surrounding the August 2010 state computer disruption.

The incident that spurred the audit occurred on August 25, 2010, and involved a failure of a key hardware component owned and operated by Northrop Grumman at the state data center. The disruption was exacerbated by subsequent delays in restoring operations tht caused a significant impact to the operation of several state agencies.   The delays in restoring the systems had an adverse effect across the Commonwealth and resulted in unexpected expense to state agencies. All told, outages related to the disruption impacted 13% of the Commonwealth’s executive branch file servers and 26 of 89 applicable executive branch agencies. The most visibly impact to the general public was to the Virginia Department of Motor Vehicles.

In the wake of the disruption, Governor McDonnell ordered that immediate steps be taken. First he directed Department of Motor Vehicle offices in every region of the state to hold unprecedented weekend open hours in order to accommodate Virginians inconvenienced by the outages. Second, he called for a comprehensive external audit of the situation, to be overseen jointly by the Executive and Legislative branches, and paid for by Northrop Grumman.   Working in conjunction with JLARC, the McDonnell Administration, represented by Secretary of Technology Jim Duffey, chose Agilysys to conduct the audit.

The audits major findings are:

  • The outage resulted from the failure of a key data storage system (EMC DMX-3) and subsequent human error during its repair.
  • Loss of data, and the delay in restoring operations and data, resulted from the failure to follow two key industry best practices.   Also, that there was an insufficient degree of self-governance towards continuous process improvement and the management of risk.
  • Many components selected for the data center and its IT infrastructure meet or exceed industry best practices, but that implementation falls short in certain areas.

Governor McDonnell issued the following statement regarding the results of the audit:

“The disruption to our state computer system last August caused the Commonwealth to incur significant expenses. It impacted a broad array of state agencies, including some of the agencies our citizens most commonly interact with on a daily basis. Many Virginians were inconvenienced by this disruption and lost hours of their time in dealing with the outages. It was an unacceptable failure and one that cannot be allowed to reoccur.

In conjunction with Northrop Grumman, the Virginia Information Technologies Agency, led by Sam Nixon, worked around the clock to bring all agencies back on line and fix the situation as quickly as possible. When that was completed, we moved immediately to take every step necessary to assist impacted Virginians.   We also called for a thorough external audit that would tell us exactly how this situation occurred and what after-action steps should take place to both address the recent outage and to prevent future disruptions of service.

The findings from the external audit conducted by Agilysys, and formally released today, confirm our basic understanding of the circumstances that led to the computer outage and subsequent delays in recovery and full restoration of agency operations.   Further, the report that has been presented provides a path forward for the Commonwealth to recover losses from Northrop Grumman for the outage pursuant to the contract between the parties.

To this end, I have directed Secretary of Technology Jim Duffey to accelerate, and bring to a prompt conclusion, discussions with senior Northrop Grumman executives on compensation and corrective actions. We are committed to holding all state contractors accountable for the performance of their duties on behalf of the Commonwealth and its citizens.”

Comments

  1. Mimi Stein says:

    Let’s see, this is the same company that has failed to live up to its original IT contract with the State, has still not fully delivered on what the sytems it promised and continues to charge the State for the budget overruns, but which last winter Governor McDonnell agreed not to sue for its mishandling of the contract and extended said contract for even more money in return for the company’s commitment to relocate its 200-person HQ from California to Virginia. Exactly how has any of this been a good deal for Virginia taxpayers?