19. Overwhelmed by Data

One of the earliest sayings I can remember is: “Don’t put all your eggs in one basket.” In the years that followed I learned the benefits of diversification and the dangers of single strand resolutions to many human problems. Now I am frightened how mindlessly we are putting all our info into one gigantic electronic basket.

But how long will the information stored by friends and family, by banks and the tax people, by Google and Yahoo, by the intelligence services as well as such social network as Facebook or Twitter remain accessible? Five years? A decade?  Most unlikely 25years. And what will this do for the  historical understanding of future generations? For now this seems of negligible concern.

Ever since the mass production of silicon chips and computers as well as the accompanying development of the internet, most of those living in the advanced technological nations have focused on both the ever-increasing speed of communications, its use in all applicable areas, and the conversion from different forms of calculation, classification, and storage to labor-saving computers. Most of the younger generation never learned penmanship and many have never written a letter!

The revolutionary consequences of the past thirty years have overwhelmed economics, markets, airline travel, research, information communication, as well as our daily lives. This has led to far more comprehensive and intensive change than the introduction of printing or even the much more gradual transformations of the industrial revolution.

From the globalization of corporations to the globalization of credit cards, the ease of  networking communications, and the immense social impact on the younger generation, the scope of this revolution is so overwhelming that it has become difficult for us to digest.  “Progress” in the form of hi-tech is moving humanity in so many new directions and at such an accelerating  speed that we are at the point of losing control.

Our taxes, bank accounts, credit card transactions, police records (including fingerprints, health records including DNA files, car licenses and driving records, CCTV recordings, NSA and GCHQ spying on our communications, are only some of the areas affecting our lives and which are now dependent on the storage of what is being called “Big Data.” Is this new information structure sufficiently resilient to cope with yet unknown threats ranging from intentional sabotage, solar flares or simply the inability to deal with the congestion caused by incredibly rapid growth? Indeed, when will the sheer volume of traffic that all these servers process become so overwhelming it causes a systemic global black-out?

The Guardian (UK), in a front-page story pegged to major computer crashes on the  Nasdaq stock market, Google, and insurance companies, warned that “governments, banks and big business are over-reliant on computer networks that have become too complex.”1

Jaron Lanier, the author-creator of the concept of virtual reality, went further in writing that the digital infrastructure was moving beyond our control: “When you try to achieve great scale with automation and the automation exceeds the boundaries of human oversight, there is going to be failure.” (Not only in the breakdown of specific servers but also of entire systems, such as the global stock exchanges and the banking system.)

And there seem to be no boundaries: Cisco the California based manufacturer of communication equipment predicts that in four years it will be possible to transmit in a three-minute burst over the internet the data equivalent to all the films ever produced.

At the same time, greed, as evidenced by the extreme demands of high-frequency computer trades of shares by hedge funds and banks, is triggering ever more mini-crashes on the stock markets. In May 2010 one such incident following a false report caused close to a trillion dollars to be erased in 20 minutes from the value of US shares. (Most of this was swiftly recovered.) Once a stock fell to an automated “Sell” level, it was impossible to halt all the other computers from executing their sell commands and across-the-board selling spread with lightning speed.

Amazon, Google, Yahoo, and such social sites as Facebook, Flicker, and Twitter, all need to process and store immense quantities of data. Twitter alone has to grapple with 500 million tweets a day! When glitches occur, and this seems inevitable, the “dirty” entities  — sometimes simply garbled or confused ones and occasionally intentionally corrupting ones from hackers or alien sources, such as North Korean or Syrian military saboteurs, can corrupt files and communications which could ultimately bring down the entire system.

At this level, there is a rising awareness of the problem. Google alone is investing around £4 billion ($6 billion) a year on network data centers. Annual global spending on such centers will rise close to £100 billion ($150billion) this year.2

The rise of big data has resulted in many traditional data warehousing companies, such as Teradata, continuously updating their products and technology. The Teradata product encompasses a massively parallel processing system referred to as a “data warehouse system” which stores and manages data. The data warehouses use a “shared nothing architecture,” which means that each server node has its own memory and processing power. Adding more servers and nodes increases the amount of data that can be stored. The database software sits on top of the servers and spreads the workload among them.3

Speed has also become a factor as has the surging demand for storage. Consider that almost three quarters of all trades on the American stock exchanges are being executed by machines which process transactions in less than a millionth of a second via fiber-optic connections. As a necessary consequence Teradata is also used as a series back-up during downtime. The systems work to balance the work-load of big data which has arisen exponentially from new media sources, such as social media.

Storing increasingly immense quantities of data also has become a challenge by placing a limit on the number of years most of it is (or can be) stored. At the same time, more steps are being taken to prevent disasters from occurring. A prime example is Microsoft’s Exchange ActiveSync (commonly known as EAS). This is a protocol designed for the synchronization of email, contacts, tasks, and notes from a messaging server to a smartphone or other mobile device. But such efforts also  illustrate how the global system has grown along market-driven choices and corporate decisions — not according to any plan nor oversight nor set rules or regulations. This does not seem like a comprehensive way to proceed globally. Alas, the possible consequences of the fantastic expansion of Big Data may not be recognized until a comprehensive melt-down suddenly occurs with the most dire results for all but the most undeveloped societies like those of Papua New Guinea or Bhutan. You have been warned!

1Juliette Garside, “Warning over data meltdown,” The Guardian (UK) 24 August, 2013, p. 1
2Charles Arthur, “The Cost,” The Guardian, 24 August, 2013, p. 7
3See the Teradata entry in Wikipedia

4 thoughts on “19. Overwhelmed by Data

  1. An absolutely fascinating read. We are in a strange new age where physical, tangible records are abandoned in favour of their more ethereal digital cousins. When I read stories of institutions like the British Library moving all of their records into ‘digital archives’, I cannot help but feel a sense of dread, that we are too readily sacrificing the written word – a medium that has proved itself over many centuries to be fundamental in the preservation of knowledge – in order that we may put all of our faith in digital media, which lets not forget without the interpretation of a 3rd party machine is completely meaningless to us. I have always been interested in the relationship between the virtual and physical needs of our ever growing digital age. People ofter consider the former without the latter, and the vast bunkers of servers, cables and hard drives that are necessary for our growing digital lifestyle to prosper go forgotten. The fact that data now sits somewhere between the physical world and the digital world while never fully occupying either strikes me as somewhat of a precarious relationship. Speaking as someone who has had more than one hard drive ‘fail’, I know all too well that all your precious digital data is still only as reliable as the flimsy and fragile piece of silicon upon which it sits. I only hope that Google don’t buy their hard drives from the same shop on Tottenham Court Road that I did, otherwise I fear that our electronic egg basket may not have the shelf life that we all think it does.

    If you haven’t already seen it, I think you would enjoy Adam Cutis’ documentary mini-series ‘All Watched Over by Machines of Loving Grace’ http://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_of_Loving_Grace_%28TV_series%29

    • Dear Joseph

      First, please excuse my rudeness/ignorance in not replying sooner to
      all the considerate, informative, and insightful messages you haver sent me.
      I still have troubles accessing WordPress and then getting my hotmail account to open…and failed today to access Twitter! (Could not send any message!)
      I think it is a consequence of insufficient experience with social mail formats.

      You spent a lot of time trying to help me and sent most thoughtful responses
      to a couple of my blogs. I particularly appreciated your confirmation of my
      concerns about the plausible evanescence of the digital media- especially
      when it comes to the need for ever more complex,(heat affected) storage
      make our current little hard-drive failures seem like a rain drop.

      The problems facing surveillance are now to be addressed by Clegg & I don’t envy him the thick web of intelligence double-talk and cover-up. Serves him right
      for aligning himself with the critique of The Mail. It truly represents the ugliest
      face of Britain!
      As to the fine points of information-trawling by future sinister figures ( or even groups) well, I agree with you- the scale would overwhelm anything Orwell
      could have imagined. All this does not bode well for coming generations.

      On more mundane matters: I have seen William Campbell Taylor’s tour
      of The City and his brave effort to enter its outer-walls, I like William
      enormously and find his honesty and directness most appealing. But
      as far as his attempted entry into politics is concerned- I regret to say
      that he is lacking both in charisma and “fire-in-the-belly.” You have to
      be really tough and driven to enter there… and those qualities do not come
      naturally to him. Nevertheless the film is visually appealing and one
      leaves it with a feeling of enormous empathy for William.

      I have taken serious note of your lengthy and insightful comments
      on my blog and the problems of making it fit into the social media.
      I am not on Facebook and am fearful of its possible infringements
      on my time and energy. I recognize that it is one of the ways one makes
      a blog more visible and appreciated… but I think I am going to have to
      hire an assistant to make that happen!

      I’m off to Paris right now to make time go backwards and admire the
      work of my father of the 1930’s and 40’s. I do hope you may get a chance to
      see it over the next few months.

      With many thanks and hope I can stay in better touch with you,

      I am a grateful,


      Reply, Reply All or Forward | More

  2. Perhaps as dire as the prospect of a “global meltdown” is the effect that the intensiveness with which we interface with data through technology has had on the way we [don’t] interact with each other.

Leave a Reply to yoricksblog Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s