Storage and data overload

Each day, an estimated 100 billion clicks are made on the web due to 55 trillion links in HTML pages. That amount of links almost match the synaptic wiring of the human brain, but unlike the brain, the web double its capacity every two years.

These days a lot of broadcasts and tutorials are available only in video format, putting further strain on bandwidth and storage needs. A large portion of the internet traffic is YouTube streaming, but unfortunately video content cannot be indexed properly by search engines (yet) and makes it harder to quickly browse the content.

If a typical email has the size of 1 kb, one small video has the size of over 5000 emails. I heard Jeff Veen say in 2008 that “Every minute of the day, ten hours of video gets uploaded to YouTube”. Half a year ago, I read that 35 hours of video was uploaded to YouTube every minute. That number has now increased to 48 hours every minute, in these few months.

The numbers are of course increasing everywhere. Facebook has reached over 700 million users and each day there are 200 million Tweets being written on Twitter. According to Go-gate, every minute over 13 000 iPhone apps are downloaded, 700 000 search queries are made on Google, 370 000 minutes of voice calls are made on Skype and 168 million emails are sent.

Dust to dust. My old GeForce graphics card in the 1990s.

Bill Gates may not have said “640K ought to be enough for anybody”, but I still remember when 1 GB was a gigantic amount of data and 1 TB was pure science fiction. In an episode of Star Trek Enterprise (aired 2001) the ship’s main computer is erased and the display read “Total volume deleted: 19.3 XB”, as in exa-byte (10^18, so 1 XB is a million TB).

The numbers will most likely continue to increase with staggering velocity, leading to a higher demand for physical storage and large cooled halls to keep the drives in. But what of the content? What will all this insanely large amount of data contain?

40 years ago, state of the art computers were used to place a man on the moon. Today we carry the same computing power in our pockets, with practically limitless potential for creativity and cooperation, but what do we do with it? Play Angry Birds.