Evolution of Big Data
In the past and more than ever today Big Data is one of the topics for modern companies as they can gain a huge benefit from analyzing their data. So, everybody is talking about Big Data, but how many actually know: “Where are the origins of Big Data and how it has developed in the past 90 years?“.
The beginning of Big Data
The recording and analysis of large amount of Data ,for that time, already started before computer even were developed. During the 1920´s criminal and arrest records could be barely kept up to date. As population growth continued to problems became worse. More and more labor had to be employed to deal with the information and the storage-systems in libraries, the original systems of storing and sorting data, had to be improved. This massive increase in information was referred to as the “Information Explosion”. Therefore, there had to be a other, more sufficient way to deal with large amount of data, other than by just employing more people.
First ideas to deal with Big Data
To be able to cope with the immense amount of data, first machines were developed to make the work more efficient and faster. One of the first was the 405 Electric Accounting Machine, which was able to process 150 80-character punch cards per minute. The machines helped to increase the speed of analysis and storing, in addition requiring less labor. However, there was still the problem of storage. As time went by machines became more and more developed and in the 1960´s first centralized computing systems were used to automate data analysis.
Revolutionary ideas were driving the development process. The concept of virtual memory, that would treat finite storage as infinite, was proposed by Fritz-Rudolf. In 1970 Edgar F. Codd introduced the idea of relational database, which showed how data could be accessed from a database without knowing how the information was structured or where exactly it was stored in the database.
The usage of just storing data was limited, therefore firms wanted to use their data to improve their decision making. Therefore concepts and methods using fact-based support-systems were used to improve business decisions. This was defined as business intelligence. Business intelligence started to boom, as a reaction to the success of the world wide net in the 90´s.The influx of information was a new challenge for data management and increased the cost for storing data. To offer more functionality, digital storage quickly became more and more cost-effective for storing data and the first BI platforms began to form.
The 21th Century
In the 21th Century Big Data was strongly promoted and had a rapid growth. Big Data stood for the 4 V´s: volume, velocity, variety and value. In addition to this, Web Services were focused to allow customers to easily create new applications. The increasing data flow due to the success of the world wide web was shown by studies in 2008, stating:“Americans consumed information for about 1.3 trillion hours, an average of almost 12 hours per day”. Therefore, Hadoop was created, which aimed at storing and processing data. The internet continued to become more and more important and first cloud solution were offered. Also other new in-memory technologies such as SAP HANA® are used until today to analyze and optimize mass quantities of data and using the analysis for competitive advantage in this very rapidly changing market in the today´s world.
So we can ask: ”What will the future bring for Big Data?”. Experts estimate a 4300% increase in annual data generation by 2020. So we can be curious to see, how we will deal with this amount of data, which is surrounding all of us.
Big Data regards