Big Data, Big news?

Big Data, Big news?

By Mark Kendal, AiM

You don’t have to look very far in the blogosphere or anywhere else these days to see the words “Big” and “Data” adjacent to each other. Fire up your favourite search engine or look on YouTube and you will find hundreds of videos, white papers and articles on “Big Data”.

Like many buzzwords in the world of computers the phrase can be, and is, defined in numerous ways.  In fact, possibly as many ways as there are people to define it. However, in essence, “Big Data” is an approach to our data that is about learning all we can from the very large amounts of information that we and our technology are generating. So “Big Data” is not about how best to store and retrieve the names and addresses of customers and staff or how to keep track of payroll data. Rather, it is about what I can learn from my data, what trends can I identify and can I predict the future behaviour of something or some people to help achieve my goals? The idea is to collect massive quantities of data then learn something useful from it.

Typically “Big Data” is machine generated in all sorts of ways – modern airliners produce gigabytes of data during one flight, computer servers create log files recording all their activity, text messages, phone calls, tweets, Facebook updates,  internet searches, apps on our smart phones that know where we are, what we are doing and what we are buying.

What can be done with all this data? A recent BBC Horizon documentary (see it on YouTube at https://www.youtube.com/watch?v=EsVy28pDsYo  ) described some of the exciting ways that this data is being used (mined). For example the Los Angeles Police Department are successfully using a mathematical model to analyse records of time, type and location of crimes. They then predict where crimes are likely to take place and plan their patrols accordingly. This way they are successfully reducing crime. Another example is a company in the City of London that accumulates information about stock and share prices from all over the world, in some cases going back centuries, and then analyses that data to identify trends and inform investment decisions. Another example, not shown on the Horizon programme, is of a U.S. university who have analysed the use of words in downloadable books. They have mapped the frequency of word use by the year that each book was written to discover how individual words have risen and fallen in use and how this reflects social and political change at national and international level.

To achieve these results a new approach to storing and analysing data is needed. Traditionally applications have been developed for a specific purpose and the choice of data to store has reflected this both in terms of content and structure. Also historically data storage was relatively expensive so no more data was stored than was absolutely necessary for the job in hand. These constraints inevitably impact on the kind of questions that can be answered by looking at this data.

Times have changed and Big Data potentially throws a great big spanner into that particular paradigm. The traditional approach that I’ve described above is excellent for processing transactions but isn’t necessarily helpful when we want to find out what is going on. Now large quantities of data can be harvested from many sources and stored using new technology designed for data analysis rather than for transaction processing. There is also a growing study and use of tools to analyse this data and identify subtle trends.

Processing transactions is different from learning from our data. There’s nothing new in that statement and the industry has been aware of this for decades. The advent of “Big Data”, its storage methods and the growing thoughts and approaches to pulling value out of that data (Analytics) is making a difference. The very powerful tools available now help us look at both our traditional and “Big” Data to aid the discovery of trends and gain more insight into what we want to know even if we don’t know what question we’re asking yet.

So let’s not throw out our SQL based databases yet. However let’s be prepared to take a fresh look at the data we currently own and the data that we might acquire to see what we can learn and what value we might derive from it.

AiM sees “Big Data” as the next phase in the information revolution to create knowledge-based economies – not a fad but something that is here to stay. We offer analytics and the ability to use data for insightful decision making both as part of our solutions and as an outsourced service line. Exciting times ahead!