Modern filmmaking techniques that consume data by the petabyte have forced post-production studios to upgrade capacity to handle the load, according to Park Road Post Production head of technology, Phil Oatley.
Based in Wellington, NZ, Park Road provides post-production services for film features, “from digital and film rushes, stereoscopic alignment, digital intermediate, foley and sound mixing through to the final completion of all film and digital deliverables for distribution,” Oatley said.
Park Road has worked on several big Hollywood films including last year’s The Hobbit: An Unexpected Journey, as well as District 9 and The Last Samurai.
“We faced a very unique challenge on one particular project, and we realised that we would need to dramatically increase our throughput and capacity to meet the potential demand,” Oatley said.
“Each shoot day would see us process an average of six to 12 terabytes of new material, and on a really busy day this could reach 20 terabytes. All new material needed to be processed and delivered to the client within 12 hours, which created a significant data management challenge for us to overcome.”
The total amount of post-production data that must be stored depends on camera resolution, acquisition and digital intermediate format choices and the amount of material shot during production, Oatley said.
“Digital cinema camera technology is constantly driving up the amount of data generated,” he said. “With digital cinema camera sensors recording at high frame rates and at resolutions upwards of 5K, the storage capacity and bandwidth challenges are immense.”
In addition, a “growing trend” toward 3D and an “emerging trend” towards high frame rate (HFR) 3D is increasing the amount of data used by a single film, he said.
“Films are commonly post-produced using a 2K or 4K digital intermediate format,” he said. “A 2K film might require as much as 0.5 TB per hour of material, and with 3D doubles this to 1TB per hour.”
HFR 3D — a new technique that was used in The Hobbit and runs at 48 frames per second — “doubles this again to 4TB per hour,” Oatley said. “Modern 3D HFR feature films can easily generate more than 10TB per day during shooting, and literally petabytes of data across an entire feature.”
Placing an even greater strain on capacity, Park Road typically works on multiple projects at the same time, he said.
After one project put a particularly high demand on Park Road’s systems, the post-production house decided it was time to upgrade, Oatley said.
Park Road had been using a traditional archive system using direct-attached libraries. The company looked for technology that would provide high performance, reliability and integrate well with its other systems, he said.
“Park Road has a wide variety of systems running on Linux, OS X, and Windows, and it was essential for any new systems to run natively on all platforms.”
The post-production house sought help from Quantum, a vendor it had used previously. “Various flavours of StorNext were already deployed at Park Road, and it was a logical step to consider Quantum's StorNext Storage Manager for virtualised tape archival,” he said.
The new shared storage platform provides 1.5PB of disk, 2PB of near-line virtualised LTO tape storage and unlimited vaulted capacity, Oatley said. “This system has been proven under some of the most demanding load conditions and unforgiving deadlines.”
The system at Park Road routinely processes multiple terabytes of data in a few hours and can handle in excess of 20TB per day at peak load.
With the film industry constantly upping the ante on visual effects, it’s possible Park Road will have to add capacity again in the future. However, Oatley said he believes the current platform will simplify any future upgrades.
“Storage requirements are always growing, and technology is always struggling to keep pace,” he said.
“Capacity will always increase, but the key is to think about the entire life-cycle of your data, streamline your workflow as much as possible, and to make technology choices that are scalable without having to re-architect from scratch.”
While many types of companies are looking at the cloud for storage, Oatley said that approach is not yet equipped to handle the needs of a film post-production house.
“Cloud solutions currently lack the necessary bandwidth to serve rich media workflows at this level,” he said.
“Cloud solutions are typically deal well with large client counts with smaller and more transactional data using a distributed model. Film post-production requires the exact opposite scenario — with fewer clients accessing massive files at high bandwidth from a central store.”
Follow Adam Bender on Twitter: @WatchAdam
Follow Computerworld Australia on Twitter: @ComputerworldAU, or take part in the Computerworld conversation on LinkedIn: Computerworld Australia