To Be Continued? CFO Magazine Interview with John Odermatt Five steps you can take to make sure your company survives catastrophe

To Be Continued?

Five steps you can take to make sure your company survives catastrophe.

Say this much for disasters: they’re educational. Some, like Y2K, may offer useful lessons in overreaction. Others, like 9/11, may remain largely incomprehensible. But most, from Hurricane Andrew to the crisis surrounding Japan’s Fukushima nuclear plant, have pushed companies to develop better response plans. In the aggregate, all of these events have advanced the discipline of business continuity, and the pace of that progress has quickened in recent years.

And not a moment too soon. Last July, reinsurer Munich Re said that 2011 had already become the most costly year on record for economic losses, due to the number of severe natural catastrophes in the first six months.

The lessons learned in the aftermath of so much tumult — along with technological changes and the increasingly interdependent nature of global business — have forced a rapid evolution in business-continuity planning. The old approach to preparedness focused almost exclusively on restoring a company’s IT capabilities. That view is now seen as far too limited. “The marker was 9/11,” says Roberta Witty, a research vice president for technology consultancy Gartner. “Until then I think most people were looking at IT disaster recovery and had never experienced an outage where the workforce itself would be so severely impacted.”

As process owners and compliance executives, CFOs can’t ignore business-continuity risks. Finance chiefs in heavily regulated sectors such as finance and health care have even more incentive to keep up with new developments in disaster planning. Here, then, are five ways that corporate planners are changing their approach to preparing for the worst.

1. The Really Big Picture
The scope of business-continuity management has expanded dramatically since 9/11. Today’s leading companies are integrating people, processes, data, and physical infrastructure into a holistic approach to business continuity (sometimes referred to as business resilience).

An international survey of 391 senior executives conducted in June 2011 by the Economist Intelligence Unit on behalf of IBM found that while only 37% of respondents had implemented an organization wide business-resilience strategy, 42% were likely to do so within the next three years. Almost two-thirds (64%) said they had a business-continuity plan of some sort.

John Odermatt, who was first deputy commissioner of New York City’s Office of Emergency Management during the terrorist strikes and was later appointed commissioner by Mayor Michael Bloomberg, has brought what he learned in the aftermath of 9/11 to his current position as head of Citi’s Office of Business Continuity: namely, that people and communications are everything in a crisis.

He’s had plenty of chances to test that conclusion, one of the most dramatic coming after the Haiti earthquake in 2010. When the 7.0 magnitude quake struck at 4:53 p.m. on Tuesday, January 12, Citi’s Haiti employees were closing transactions for the day. Suddenly, the company’s three-story headquarters collapsed around them.

Workers who weren’t buried in the rubble struggled to make sense of what was going on. One of them was able to contact Citi’s regional crisis-management team in Mexico before telecommunications went down. Immediately, the Citi crisis-management team activated its command structure, coordinating the company’s response. Citi security helicoptered in security personnel to help rescue employees and transport the injured to the Dominican Republic. Within a week the team was delivering humanitarian supplies to the area, eventually providing 15 tons of aid, including satellite phones.

Tragically, 5 of Citi’s 43 Haiti employees were killed in the earthquake. But the company’s response was deemed critical in providing care for other workers and getting the business back up and running quickly.

Transactions from the day of the quake were cleared abroad, and when banking in Haiti resumed 11 days later, employees were ready to operate at a shared site. “From a people, humanitarian, and business perspective, everything and anything that was asked for was coordinated through our central team,” says Odermatt. “I think that’s the secret to any successful recovery.”

Of course, military-style logistics like that don’t happen on the fly. Citi’s arrangements were well established and practiced before the Haiti earthquake. In the chaos of the tragedy, the preparations allowed employees to understand where to go and how communications should flow.

Throughout the world, every line of business at Citi is involved in continuity planning. Rigorous testing and crisis planning involve everyone from the CEO down and occur at every level of the organization. “In addition, there is joint industry testing where the markets make themselves available so we can test our technology on nonproduction days,” says Odermatt. “I think such testing is one of the things that set the financial industry above other industries.”

2. Public-Private Collaboration
A decade on, one legacy of the 9/11 attacks has been to highlight the interdependence of the public and private sectors. “Governments realize that a large portion of public services is provided by private enterprise, so government is very dependent on business,” says Gartner’s Witty. “And private enterprise is starting to recognize that without first responders — the police, road crews, and government — you can’t do anything.”

The Federal Emergency Management Agency (FEMA) created an entire division devoted to public-private partnerships in 2007. The division nurtures engagement with businesses and provides helpful tools, such as downloadable tabletop exercise materials and a free online course in public-private relationships (see “Some Help from the Big Boys,” Topline, September 2011).

At the local level, liaisons in all 10 of FEMA’s regions are developing relationships with community businesses to facilitate resource and information exchange. In an emergency, FEMA and local emergency officials have developed procedures for determining the status of utilities, communications, medical facilities, and food and supplies, for instance. They can then feed that information back to local businesses, letting them know about critical developments such as when power will be restored. In turn, businesses may have resources to share, such as disaster hygiene kits or parking lots that can be used for emergency operation centers.

The public-private collaboration “has taken off like wildfire,” says Dan Stoneking, director of FEMA’s private-sector division. One of the large companies that is working with FEMA is Verizon Wireless. The partnership aims to provide communications aid to disaster-hit areas. As part of ongoing preparations, a Verizon Wireless technician participated as a temporary FEMA private-sector employee for three months, giving the company firsthand insight into what goes on inside the agency. “It also gives us a gut check on how we do our job,” says Stoneking.

The telecom giant also teams up with state and local governments and nonprofit entities that support emergency responders. More than 45 Verizon Wireless crisis- management teams are dispersed across the country to respond to local needs, while a central team and hotline coordinate requests for emergency wireless voice and data products or wireless network support.

Requests may come from, say, the American Red Cross for 20 loaner mobile phones, or from officials in remote locations needing what Verizon Wireless refers to as a “cell on wheels.” “We have these mobile assets that we can deploy to help agencies set up mobile command centers without which they really could not operate as effectively,” says Gabe Esposito, Verizon Wireless’s director of corporate security, business continuity, and disaster recovery.

3. Shoring Up Supply Chains
Hurricane Katrina in 2005, the 2010–2011 floods in Australia, and particularly the earthquake and tsunami in Japan last year have all emphasized the vulnerability of international supply chains. “While companies were able to recover their operations [following the disasters], they may not have been able to get the components they needed to restore manufacturing,” says Greg Bell, a partner at KPMG. “People have been thinking about everything from, ‘Do I need more supplier diversity for my key parts?’ to, ‘How do I get visibility into my suppliers’ business-continuity plans?’”

Goodyear, for one, has been examining those questions extensively. When the Sendai earthquake struck last March, the tire maker’s operations in Japan escaped relatively unscathed. But many second- and third-tier suppliers to the auto industry were affected. That’s when Goodyear’s provisions for alternate sources and intraplant transfers kicked in. “Through a robust supply chain, there’s a possibility you could have necessary materials in another location,” explains Mike Janko, Goodyear’s manager of global business continuity. “It may cost a premium to ship it, but the end goal is to make sure we’re meeting the needs of our customers.”

A vendor partnership program that Goodyear began before the quake now seems all the more prescient. The company estimates that 15% of all the crises it deals with are related to product-supply disruptions. With that in mind, business-continuity managers joined with the purchasing department to determine which of the company’s hundreds of global suppliers would have the biggest negative impact if something went wrong. They pinpointed about two dozen raw-material suppliers in the first round, and the continuity team is now working with them to beef up resiliency planning.

For many companies, however, it’s not just raw materials that are in question. Outsourced services from managed data centers or technology providers raise concerns. The financial industry, for instance, deals with clearinghouses throughout the life cycle of transactions. “Many companies, including Citi, outsource their services and do enormous amounts of offshoring,” says Citi’s Odermatt. “There’s more-intense focus now on what those suppliers’ supply chains are, what their business-continuity plans are, and whether they’re being tested.”

4. Virtually Bulletproof
In the data center, virtualization has been lauded as a boon for business-continuity planning. In this technology, multiple virtual machines — consisting entirely of software, each using a different operating system and running a different application — can run independently on one server. That means fewer hardware boxes are needed to run the same number of applications, and those boxes are each more efficient. While regular servers normally run at only 5% to 15% capacity, a server running virtual machines can operate at 60% to 80% capacity.

Because virtual machines are independent of the hardware they run on, they can be easily moved around a firm’s network or to any other server deemed necessary. Copies can be saved offsite for disaster-recovery purposes.

The Texas Association of School Boards had those benefits in mind when it rebuilt its data center three years ago using virtual machines. The agency, which provides insurance, workers’ compensation, and a purchasing cooperative to more than 1,300 school boards, needs 24/7 IT service. When it began the virtualization process, only 8 of its 100 applications could be recovered from a mirror site after a disaster. Now, 94 can be brought back up within 15 minutes.

For system administrator Toni Fowlie, however, the project generated new problems. When wildfires swept the central Texas area last summer, her concerns about the heat outside were minor compared with what was going on in the data center. “I didn’t worry so much about the fires, but I do worry about power,” says Fowlie.

The reason is the blade servers that run the Texas agency’s virtual machines. Although the thin, stripped-down servers are more energy-efficient than their predecessors, more units now fit into the same space, which strains the data-center infrastructure. “Power-to-performance is greater, but you’re performing more and you’re condensing more,” says Mark Vanston, director of business continuity and recovery services for HP Enterprise Services. “In the past, [data centers] were designed to handle a certain power load per square foot. Years ago that was probably 80 kilowatts. Now you need about 150.”

Cloud computing, while still in its infancy, could alleviate some of these headaches, but will likely also raise new ones for disaster-recovery managers. Not only will they need to worry about the viability of their cloud suppliers, they will also have to create contingency plans regarding Internet connectivity to those suppliers.

5. All Together Now
For crisis communications, a new, democratic order is at hand. Social media has changed things forever. “Social media is not just a new way to broadcast information,” says John Orlando, a social-media consultant. “It reverses the direction of communications.”

Researchers from the universities of Colorado and California at Irvine found that during the Southern California fires of October 2007, residents turned away from mass media and official sources of information and looked to peer-to-peer resources such as blogs, community forums, e-mails, text messages, and Twitter to find out whether a fire was headed down their street. These outlets provided better, more-timely information, as well as the means to disseminate it. In many instances, the participation of community members helped keep rumors in check and validate information from reliable sources. Respected community sites like rimoftheworld.net collaborated with fire departments to post up-to-date news, which was then reposted on other local forums and discussion boards.

“Emergency managers have to understand that the public is going to self-manage the disaster with or without them,” says Orlando. “So the challenge is to develop a collaborative model where the old assumption that the public is a problem to be managed is replaced with the assumption that the public or your employees are a resource to be harnessed.”

The private sector has been slower on the uptake, but it is beginning to use social media to converse with customers during emergencies. TD Bank used its existing Twitter program to monitor consumers’ concerns during Hurricane Irene. When questions about available ATMs and branch closings cropped up, the 10-person Twitter team responded with updates and links to mobile apps showing available facilities.

Still, corporate social-media programs to communicate with employees during emergencies are but a future vision for most companies. Orlando suggests that corporate Facebook pages could be used by employees of companies that have been shut down by a crisis. “Instead of just communicating information outward, it can be a way for people to coordinate their needs with one another,” he explains. “They can ask, ‘Can someone pick up my daughter from school?’ or, ‘Could someone pick up groceries for me?’ And by [providing] a place where they can form a community and call on one another, you’ll make it much more likely that they’ll be there for you when it’s time to start up business again.”