The Cost of Failure

By Maria Trombly | Securities Industry News | November 14, 2005

There are times when a business-continuity disaster is completely unforeseen and cannot be planned for. And in other cases it is clear that the people in charge should have known better and planned specifically for such contingencies. That's a lesson that seems to have to be learned repeatedly in the world of stock exchanges--most recently in Tokyo.

There are two leading causes of stock exchange failure: power or telecommunication outages, and software upgrades. These happen to be the most easily avoided disasters because of their predictability. The probabilities of electrical blackouts, for example, are measurable and common knowledge among private-sector contingency planners, utility executives and public officials. A stock exchange that does not have a contingency plan for this occurrence is not working with a full deck. One might assume that after the Sept. 11, 2001 terrorist attacks or the August 2003 power outage in the Northeast U.S., no major financial market would have failed to take heed. But this past March, the London Stock Exchange (LSE) suffered a power outage that left some customers unable to see market data or trade for several hours.

With instant communications shrinking the globe, any problem at a given stock exchange is immediately transmitted around the planet.

Disasters are hardly rare around the world; the U.K. has faced more terrorism than the U.S., for example. And this summer's bombs in London's transit system didn't stop trading at the LSE. Yet European exchanges have shown some vulnerabilities, perhaps because they have been less prodded by their regulators to institute business continuity planning to the extent that their American counterparts have.

One solution is for the regulators to step in and force exchanges to make and test backup plans. U.S. agencies led by the Federal Reserve Board and Securities and Exchange Commission have done just that, and the financial industry and its trade associations have purposefully fallen into line. By the time of the 2003 power-grid collapse, there was virtually no financial markets disruption.

Japan, meanwhile, saw two outages in the course of a single week this month, and regulators there, right on cue, are now forcing a good, hard look at disaster preparedness. The issue this time was software upgrades. Averting breakdown seems all too simple: Just do nothing. Don't upgrade the software, and you're okay. Break what isn't broken, and you have a problem. The rallying Tokyo Stock Exchange (TSE), however, had been facing rising share volumes--a 70 percent increase between July and September alone--and felt it had little choice but to upgrade its systems. The crash on Nov. 1 took the market out of commission for four and a half hours, preventing trading in cash securities for all but 90 minutes of the daily session.

Last week, the TSE took a big step toward holding accountable those responsible. It cut the pay of exchange president Takuo Tsurushima and chairman Taizo Nishimuro in half for six months, while systems chief Tomio Amano suffered a 30 percent cut. Seven other officers got 10 percent to 20 percent penalties. The cause of the outage was traced to new software from Fujitsu, which has said that it may also penalize its executives.

Japan's Financial Services Agency has ordered a review of IT systems at all six of the country's stock exchanges: the TSE, Osaka, Nagoya, Sapporo, Fukuoka and Jasdaq. (Jasdaq, Japan's board for start-up companies, has also been plagued by systems outages. In August, trading was halted for three hours, following two other instances earlier in the year.) Reports are due by the end of this month, according to financial services minister Kaoru Yosano.

The official investigation is aimed at not only identifying the reasons for the outages, but also at preventing recurrences, Yosano said.

Those lessons, though, are pretty well learned already from experiences elsewhere. Software, whether new or in upgraded versions, needs to be tested in controlled environments before being deployed live. Systems also need to pass scalability and stress tests to meet the challenge of rising volumes. Disaster plans must not only be tested, but must be ready to be implemented instantaneously, not over a period of several hours. A good plan should include "rollback" options, so that systems can be restored to a previous, working state in case an error in a software upgrade slips through the testing filters. Finally, system components need to work independently of one another so that if one goes down, others can continue to function.

With instant communications shrinking the globe, enabling reliable, round-the-clock connectivity among market players and centers, any problem at a given stock exchange is immediately transmitted around the planet. Given the prevalence of cross-border trading, any glitch can affect securities firms everywhere. It's no longer just a local embarrassment, and it can escalate into an epic public relations nightmare. At the same time, global interconnectedness also offers exchanges and regulators more opportunities to learn from each other's problems and errors, out of which best practices can--and must--be developed and shared.