Storage and data overload

Each day, an estimated 100 billion clicks ricochet around the web, following some 55 trillion links buried in HTML pages. That’s roughly in the neighborhood of the number of synaptic connections in the human brain.

But unlike the human brain, the web apparently feels compelled to double its capacity every couple of years. And we keep feeding it.

More broadcasts, lectures and tutorials are appearing exclusively as video, because apparently reading was becoming dangerously efficient. YouTube alone consumes an enormous amount of bandwidth and storage. Worse, video remains difficult for search engines to properly index, turning what might have been a paragraph of useful information into a fifteen-minute archaeological expedition involving buffering, introductions and somebody saying, “Hey guys, welcome back to my channel”.

Consider the numbers. A typical plain-text email might occupy around one kilobyte. A small video can easily consume the space of thousands of those emails. Back in 2008, I heard Jeff Veen say that ten hours of video were being uploaded to YouTube every minute. Half a year ago, that figure had become 35 hours. Now it’s 48 hours. Every minute.

By the time you finish reading this, several days of footage will have been uploaded. Much of it presumably featuring cats, shaky concert recordings and people explaining things that could have been explained in three sentences.

The numbers are of course increasing everywhere. Facebook has reached over 700 million users and each day there are 200 million Tweets being written on Twitter. According to Go-gate, every minute more than 13 000 iPhone apps are downloaded, 700 000 Google searches are performed, 370 000 minutes of voice calls are made on Skype and 168 million emails are sent.

Human civilization, it seems, has finally constructed the greatest information network in history. Mostly so we can tell each other what we had for lunch.

Dust to dust. My old GeForce graphics card in the 1990s.

Bill Gates may not have said “640K ought to be enough for anybody”, but I still remember when one gigabyte sounded enormous. A terabyte belonged somewhere aboard a spaceship, preferably accompanied by blinking lights, sliding doors and a nervous engineer announcing that the main computer was about to explode.

In a 2001 episode of Star Trek Enterprise, the ship’s computer is erased and a display reports: “Total volume deleted: 19.3 XB”. That’s exabytes. One exabyte is a million terabytes (10^18). At the time, it sounded suitably futuristic. Today it sounds more like somebody describing next year’s data center.

The numbers will continue climbing. More disks. More servers. More enormous refrigerated buildings humming quietly on anonymous industrial estates, keeping our photographs, emails, videos, status updates and forgotten PowerPoint presentations alive long after anybody remembers why they were created.

But storage isn’t really the interesting question. The interesting question is: what are we filling it with?

“As I’m writing this, I’m drinking tea (made with leaves shipped through a supply chain more than three thousand miles long) out of a glass bottle (smelted at a temperature unobtainable by humans not long ago), and I’m working on a computer that would have cost a million dollars ten years ago, except you couldn’t buy one at any price, and the computer is connected to the Internet via Wi-Fi (it’s all a miracle). “
— Seth Godin, The Icarus Deception

Not long ago, state-of-the-art computers helped put human beings on the Moon. Today we walk around carrying machines in our pockets with vastly greater computing power, permanently connected to a planetary communications network offering almost limitless possibilities for knowledge, creativity and cooperation.

An extraordinary achievement. A miracle, really. And what do we do with it? We play Angry Birds.