Petabyte Hard Drive

The 2009-2010 blockbuster Avatar, which won acclaim for its special effects as well as a meteoric run at the box office, was and still is the largest movie ever rendered, in terms of computing power. It is interesting to analyze the processing power necessary for such an undertaking. According to TheNextWeb.com:

“The entire rendering of Avatar reportedly requires over 1 Petabyte of storage space according to BBC’s Clickbits, which is the equivalent of 500 harddrives of 2TB each. That’s equal to a 32 year long MP3 file(!).

The movie was shot and rendered in Miramar, New Zealand, (population 8,334) in one of the most powerful purpose-built data centers in the world (same place they created King Kong and Lord of the Rings). According to Information Management, the computing core – 34 racks, each with four chassis of 32 machines each – adds up to some 40,000 processors and 104 terabytes of RAM. The blades read and write against 3 petabytes of fast fiber channel disk network area storage from BluArc and NetApp.”

As petabyte processing becomes more and more prominent, will we see more movies like Avatar? Will the rendering process become quicker and quicker, so that complicated CGI effects such as those seen in Avatar can be created in hours and days, not weeks and months? Only time will tell, but it appears to be safe bet that given Avatar’s success both with film critics and box office customers, that we’ve only just begun.

A hard drive able to hold a petabyte of data, a longtime exercise in imagination for science fiction writers and computer enthusiasts, has actually been reality for some time. Though still largely unavailable for consumers on the mass market, and currently only in use by a select few corporations, petabyte hard drives are rapidly coming down in price and experts predict that in only a few short years, they will be standard fare for computer users.

What can I do with a petabyte?

Many of us will immediately think of recreational uses for petabytes of storage – storing massive amount of movies and music, rendering graphics in seconds, and running realistic video games. However, in addition to leisurely pursuits, a petabyte of processing power and storage will allow for an amazing amount of new scientific breakthroughs. For example, as a petabyte hard drive approaches the theoretical limit of human memory, computer applications should be able to shed more light on neurological functions. Being able to replicate a human mind will help in the fields of artificial intelligence as well as helping to perfect the long-term project of mapping the brain.

Another example involves large scale text mining and natural language processing. With the sum total of human knowledge estimated to be a few thousand petabytes (and as a reference point the entire collection of the Library of Congress is estimated to be only three petabytes), many interesting textual analyses could be quickly undertaken. For example, one could instantly compare every best-seller throughout the history of literature to see where themes and influences overlap.

Finally, scientific endeavors which rely on cloud computing, or sharing processing power across many computers, should be able to work faster and accomplish more. For example, SETI (the Search for Extra-Terrestrial Intelligence) has famously used the help of untold computer users running their search programs during downtime in their quest to document any evidence of alien life. The availability of computers with petabytes of memory and processing power will help have an astounding effect on their endeavor.

When will we see Exabyte hard drives?

It could be a long time, experts predict. Though the technology is theoretically capable of producing such a storage device, it would need a gargantuan processing speed, take up an enormous amount of space, and consume a dangerous amount of environmental resources. For these reasons, the concept is largely theoretical and it will be some time before Exabyte and yottabyte hard drives are available, though according to Moore’s Law, such technological marvels should make an appearance in the average computer user’s lifetime.