In Holland, during the winter of 1636/1637, the price of tulip bulbs rose dramatically. By February, the price of certain bulbs was equal to the price of a large house. Then suddenly, the price of bulbs began to drop sharply; by May, they were practically worthless. Many people were wiped out financially, and the country was shaken.
In the United States, during the roaring 20’s, the stock market boomed. This caused great excitement, and people from all walks of life were ‘in the market’. In October 1929 the market crashed, and by July 1932, the market had lost 89% of its value. The collapse of the banking system and the great depression followed.
During the dot-com boom of the late 1990’s there was great excitement over the creation of ‘new economy’ companies based on the internet. New internet companies, some without profits or revenues, were funded and went public. The prices of these dot-com stocks soared. The NASDAQ index (which was heavily weighted with technology and dot-com stocks) peaked in March 2000; it then began to decline precipitously, and by October, 2002, it had lost 75% of its value. In the market crash of 2000-2002, $5 trillion of market value was lost.
As the real estate market boomed in the late 1990’s and early 2000’s, people bought houses that they couldn’t afford, banks progressively lowered their lending standards, and investors bought financial instruments they didn’t understand. It was widely believed that rising real estate prices and the pooling of mortgages in new financial instruments eliminated risk. Then in 2006, real estate prices began to decline; by 2008, millions of mortgages were in default, and the value of the new financial instruments declined precipitously. By the fall of 2008, the entire financial system was on the brink of collapse, and the worst economic downturn since the great depression ensued.
In all of these speculative bubbles, ‘irrational exuberance’ was present. People behaved as if they were under the influence of a ‘reality distortion field’. They were enthralled by a compelling narrative — people will come from all over europe to buy tulip bulbs, or the internet made traditional financial analysis irrelevant — which blinded them to the actual situation. Information which seemed to support the ‘narrative‘ was emphasized, and anything that conflicted with it was explained away or ignored.
Now we can again witness ‘irrational exuberance‘. This time surrounding ‘big data’. There is again a compelling narrative: data and algorithms will transform human life. Any claim, no matter how extravagant, is taken seriously:
- Chris Anderson wrote a 2008 Wired Magazine article titled ‘The end of theory: the data deluge makes the scientific method obsolete’.
- A 2013 book titled ‘Big Data – A Revolution That Will Transform How We Live, Work, And Think’ trumpets the rise of correlation, and the decline of causality.
- In a 2014 Mashable.com article, Linda Burtch is quoted as saying: “Within 10 years, if you’re not a data geek, you can forget about being in the C-suite”.
While the claims for ‘big data’ are extreme, supporting evidence is sketchy or nonexistent. Few examples of successful ‘big data’ projects outside of internet marketing are given, and no theoretical basis is offered. A 2/2/15 article in InformationWeek reported on a survey of executives which found that only 27% of ‘big data’ projects were successful. This lack of success was attributed to faulty implementation; possible limitations to the technology were not considered.
Of course, companies offering ‘big data’ services are happy to promote the ‘narrative’ through marketing hype. And interestingly, one proponent of ‘big data’ said “it’s probably a bubble”, and another said “big data has certainly been hyped”. They both then continued to hype ‘big data’!
So, will it be different this time? Will the use of ‘big data’ grow without limit? Are you planning to take a job with a ‘big data’ startup company?