Old water tank houses hurricane-proof data center

After dismantling the city's data center and moving it three times to avoid hurricanes, the IT team for the city of Altamonte Springs, Fla., decided to try a different approach. Instead of spending millions of dollars to build a facility that would keep water out, they relocated the data center to an existing structure that was originally designed to keep water in -- a 770,000-gallon water tank.

Larry DiGioia, director of information services for the central Florida city of 45,000, says the move made perfect sense. The dome-shaped tank offered 8-inch-thick walls of reinforced concrete and was situated only 100 feet from City Hall, where a single server room housed the city's previous data center.

Along with vendor partners, DiGioia and his 10 IT staffers built a completely virtualized environment, dramatically increased network throughput, and reduced the physical server count and storage-area network (SAN) management requirements.

Compared with the old setup, the new infrastructure offers improved uptime and superior disaster recovery capabilities.

Anthony Apfelback, the fire marshal and building official for Altamonte Springs, says the upgrade not only eliminated recurring intranet downtime, but also allowed his department to roll out several new applications that automated the permit-issuing process, reducing labor costs.

"Before, it was constant downtime. We would lose our network system for a day and sometimes two days," he says. "Since Larry came in, it's been night and day. Every aspect of the system has improved dramatically. We've had no downtime."

Bob Laliberte, an analyst at Enterprise Strategy Group, says that for state and local governments strapped for cash, using existing infrastructure or sharing facilities with surrounding communities is a practical step. "Everyone's looking to find newer, greener ways of building data centers and looking at the natural landscape to do that," he says.

Altamonte Springs isn't the only place where you'll find data centers that take advantage of existing or natural structures. Information management service provider Iron Mountain planted its main data center 22 stories underground in an abandoned limestone mine, for example. Data centers have also been located in shipping containers and former nuclear bunkers -- there's even one in an old bomb shelter under Uspenski Cathedral in Finland.

A Rocky Start

The water tank idea had its merits, but the city's journey to an optimized data center with a solid disaster recovery infrastructure didn't begin easily. In 2003, only three days after DiGioia moved from New York to Florida to start his Altamonte Springs IT job, the city's network engineer walked up to him and said, "We've just lost everything. All the Novell servers, the Novell clusters, the backup, the SAN. Everything's gone," he recalls.

The city's only backup consisted of a server running Veritas Backup Exec to a Spectra Logic AIT-3 tape library. "Tapes are unreliable," DiGioia says. "Disaster recovery was nonexistent. It consisted of backup tapes in a box."

Over a two-and-a-half week period, DiGioia says, he was able to recover most of the city's data off the backup tape, but a significant amount was lost.

Things didn't get better after that. In 2004, Altamonte Springs was hit by Category 3 and 4 hurricanes Charley, Frances and Jeanne. City Hall and the data center within it were constructed to resist Category 1 hurricanes. To safeguard the IT equipment, everything had to be packed for each storm and placed into storage units until the storm passed.

"We literally had to dismantle everything. It was a horrible experience. The emergency operation center was shut down also because there wasn't infrastructure in place to support Internet access during a storm," DiGioia says.

He decided it was time to build a better infrastructure, one that would include a separate disaster recovery facility, keep critical backup data online, and support recovery point and recovery time objectives.

The water tank just seemed like a logical site, he says. It had been decommissioned a few years back and was built like a fortress.

First, DiGioia got buy-in from Altamonte Springs' political leaders. The city then commissioned the construction of two buildings on either side of the water tank; one wing now houses the networking equipment and the other has administrative offices.

At a Glance

* City of Altamonte Springs, Fla., an Orlando suburb with about 45,000 residents

* Project champion: Larry DiGioia, director of information services

* Size of IT department: 10 staffers

* Project payback: Converting a water tank into a data center cost the city less than $2 million and took a little more than a year and a half. The team finished last fall. The two SANs cost $250,000; the VMware deployments cost $5,000 per server, or $60,000 total; and refurbishing the water tank cost $1.5 million. The new data center is hurricaneproof and has eliminated network downtime.

In the old data center in City Hall, the networking infrastructure consisted of point-to-point T1 lines over copper wire to 16 facilities, such as the police department and public works offices. For the new data center, the city ran dark fiber (unused optical fiber that can be tapped into if necessary) to the same facilities; it also leased dark fiber from surrounding county governments, greatly increasing bandwidth and distance for disaster recovery.

DiGioia and his team also rolled out VMware ESX server software and reduced the physical server count from 80 boxes to 12 Dell quad-cores running 30 virtual machines. The servers were configured to boot from the SAN.

In the main data center, the city uses CommVault Simpana software to back up to a Xiotech Emprise 7000 SAN. Xiotech's TimeScale continuous data protection (CDP) system is then used to replicate that SAN to another Xiotech 7000 SAN at the city's disaster recovery site. Each of the SANs has 25TB of capacity. Xiotech's ICON Manager interface gives the IT team a single point of management for all of its storage activities.

"The SAN is mirrored, everything's virtualized, we have CDP," DiGioia says. "It's just a night-and-day experience."

Backups are kept on disk for 30 days and then overwritten, and tape is no longer used. Documents are archived on optical disc and microfilm.

Choosing a Partner

DiGioia says he selected Xiotech Corp. as his SAN vendor because its previous service on technology the city owned had been outstanding, and other vendors, including Hewlett-Packard and Dell, didn't seem interested in an operation as small as his. DiGioia says he also looked into using NetApp but felt it couldn't offer the level of service Xiotech could.

DiGioia says Xiotech demonstrated that it was "interested in the long haul" rather than "a hit-and-run" at Altamonte Springs. "Xiotech was extremely responsive to us. I have access to the right people," he says. "I've never had a problem with the device as a whole, and everything was always a smooth transition with them."

In order to meet recovery objectives for city's various departments, DiGioia met with each of them, determining their individual needs and tweaking systems to meet those needs.

The new, more highly automated infrastructure enabled the city to set up an online IT help desk, which is able to provide instant feedback on department service requests along with estimated times for ticket completions. "Before, we'd have problems and submit them and they'd get lost," Apfelback says.

The new data center also meant the fire department's and building office's Web sites and intranet would no longer experience three-to-four-day outages every month. With the increased bandwidth and reliability, DiGioia's office was also able to launch a new online building-permitting and inspection request system. Handling 1,100 to 1,400 inspection requests and up to 250 permit inquiries each month, the new online application system has streamlined how customers submit and track their requests.

"For our customers, they can go on online 24/7, 365 days a year and request inspections and see the status of their inspections," Apfelback says. "It's really brought us into the next century and where we need to be from a customer service standpoint, both internally and externally."