Get IT Done: Evaluate data centers as part of a disaster planning and recovery strategy
Published on January 28, 2004
on CNET's TechRepublic.com
Companies with mission-critical data systems and near-zero tolerance for downtime must have disaster recovery measures in place, often including one or multiple external data centers. But selecting a cost-effective data center with five nines of availability (or 99.999 percent uptime) is wrought with challenges. With so many possible points of failure at a data center, CIOs need a guide to help them evaluate collocation providers.
The following report highlights key questions CIOs should ask when evaluating data centers, with advice gathered from three top data center experts: David F. Locke, contingency planner with San Mateo, CA-based financial services company Franklin Templeton; Lou Kirchner, president and CEO of Sacramento, CA-based data center Herakles LLC; and Ron Hughes, president of California Data Center Design Group, who has been involved in the design, construction, and operation of data centers for over 20 years.
Outsourcing: A low-cost alternative
With the cost of building your own data center running upwards of $1,000 a square foot—well out of reach of many enterprises even if the construction expense is amortized over a 20-year period—some companies opt for external data center providers. These providers once offered facility leases for as much as $150 a square foot per month, according to Hughes. Because many data center providers built facilities under faulty financial and market assumptions, several have been forced into bankruptcy and sold for pennies on the dollar. The new owners can now provide rock-bottom data center leases priced at $8 to $12 a square foot per month.
Low prices, however, can't be the only measure when selecting a collocation center. After evaluating their needs from a data center, CIOs should consider the following:
When selecting a data center provider, one of the first things to consider is the geographic location of the data center itself.
- How close does the staff need to be to the data center in the event of emergency?
- How quickly can staff get to an external location via alternative forms of transportation in the event that airplanes are grounded?
- Is the data center located in a geographic threat zone, a place prone to seismic disturbances or severe weather, such as hurricanes, or is it located on a 100-year flood plain? (All of these pose a real threat to a data center's uptime.)
In an evaluation of a facility's design, it's crucial to look for any single point of failure. Examine building construction plans with an eye to determine if the site can be operated and maintained without an interruption in service.
- Is the facility designed for five nines of availability?
- Is there adequate redundancy and reliability designed into the systems?
- What types of mechanical systems are installed?
- In terms of air-conditioning units, does the facility use chilled water or air cooled units? (Air cooled units are less efficient and drive up facility costs and your end price.)
- Are there redundancies built into the facility's electrical switches that power the floor? (If the electrical system for the building shorts out, no form of electric power generation will fix the problem.)
- Are there at least two fiber carriers entering the building?
- Are these carriers entering the building in different fiber vaults?
- What is the present electrical load in the facility?
- What capability does the facility have to support itself if power from the main grid goes down?
- How many backup generators exist? (There must be at least two generators, each one with the capacity to generate enough electricity at full load to power the facility.) Has the company used best-of-breed data center practices as defined by associations versed in business recovery and
- emergency planning, such as BRMA or AFCOM?
- Has the infrastructure been tested in a real emergency or simulated emergency?
Security of the data center itself is an important consideration. For that reason, before examining the inside of a data center, start on the outside by asking the following types of questions:
- Does the facility stand alone or does it share a building with other tenants?
- Do other tenants pose any kind of threat to security (direct or indirect competitors, many people entering and exiting, frequent shipments of products)?
- Is the site surrounded by fences, berms, or shrubs that pose easy access points to and from the perimeter?
- Is the parking lot gated with guards at the checkpoint?
- Are the security guards professionals employed by a security firm or are they merely employees of the data center? (Data center employees may not be properly trained.)
- Are the security guards present 24 hours, seven days a week?
- Are there adequate security cameras covering the perimeter and entrances?
- Is activity on these cameras being recorded?
- How long are video surveillance tapes archived?
- Does the building have controlled access with biometric security?
In terms of operations, look for any single point of failure and don't overlook things that seem small or insignificant. An overlooked detail, such as a poorly designed fluorescent lighting circuit can cause a power panel to trip, leaving an entire facility without electricity. This particular scenario may sound far-fetched, but one of Franklin Templeton's data centers experienced just that, according to Locke. Locke's team traced the problem back to a failed fluorescent lighting ballast, a kind of transformer.
Insufficient maintenance of the facility's power generators or air-conditioning units might cause downtime, too. The risk of failure rises as data centers attempt to save money by trimming maintenance schedules. To begin an evaluation of the operations, examine the following:
- Who has actual accountability for operating the facility?
- How often is maintenance of air conditioning, heating units, power generators, and other systems performed? (Ask to see the records.)
- In the event of a failure in the air conditioning, power generator, or other system, are there sufficient spare parts on site to fix a variety of common or likely problems?
- How often are backup generators tested?
- Are the backup generators tested under full load to simulate real power demands for several hours?
- What expertise does the support staff have, and is the same level of support available every hour of every day of the year?
Many collocation centers built in the last decade were developed under business assumptions that proved to be unrealistic. When the Internet went bust, the predicted demand for collocation centers failed to materialize—and so did the revenue. Many facilities were put on the auction block while others cut back on maintenance and staff in an effort to remain solvent. Under these circumstances, companies interested in leasing space at a data center would be wise to look at the provider's financials as well.
Ask these questions:
- Was the facility sold in a fire sale, making it easier for new owners to actually make a profit?
- Is the company profitable?
- What are the operating expenses?
- What are utility costs and maintenance costs?
- Do the operational costs, including full-time staffing of engineers, security, maintenance, and utilities, seem realistic, or does it appear that the company is trimming corners somewhere?
A final precaution
Finally, even after a thorough evaluation of the preceding points, if the facility passes muster, it doesn't hurt to bring in a consultant or interview the original designers of the facility. Performing the preliminary investigation with your own staff helps to familiarize them with a facility and can save on consulting costs. But external expertise can assure top management that the facility really has no single points of failure, says Locke. After all, when the servers are installed and the data is mirrored, a failure in the data center can be an embarrassing "gotcha" that no one wants to experience.