Introduction and staff issues

Big data this, big data that. It’s talked up as a business revolution, and has caused many a boring PowerPoint presentation, but are businesses actually using – and benefiting from – big data?

According to recent pan-European research into the impact big data has on organisations, over half of businesses have missed opportunities they didn’t see coming because they lacked accurate information at a time when they really needed it – and it’s cost them as much as £20 million (around $29 million, AU$41 million) per year.

What’s more, Pure Storage’s research showed that over 72% of businesses admit to collecting data they never use, calling it time-consuming and expensive to process.

“The reason we’re seeing these trends emerge is because it is now cheaper for businesses to retain the data they are collecting, than to destroy it – so the volume of data a business holds is growing rapidly,” says James Petter, VP EMEA, Pure Storage. “But at the same time, it is complicated and costly to access usable information fast enough to make a difference.”

So is big data going to waste?

James Petter, VP EMEA, Pure Storage

Are companies making the most out of big data?

Business transformation consultancy Moorhouse recently commissioned some independent research into FTSE250, UK public sector and UK-headquartered multinationals that showed only 11% of such organisations believe they are leveraging big data effectively to inform their strategic decisions.

“First the industry promised untold benefits, we then moved on to scepticism and we’re now at the point of seeing the first practical wins,” says Matt Davies, Head of EMEA Marketing at Splunk. “We’re only at the beginning of the big data story and the next couple of years should see the industry focusing on the value and delivering on that early promise of big data.”

Giles Slinger, Director at data analytics firm Concentra

Is big data all talk?

It’s a buzz-phrase, it’s a popular hashtag on social media, and it looks great on a CV. Big data is so ‘zeitgeisty’.

“IT directors and marketing directors face growing corporate peer pressure to invest in big data,” says Laurence Armiger, Sales Director at Zizo. “Some IT directors are dabbling with big data because they want to have it on their CV – ‘I’ve put two million records into a Hadoop database’ etc. – not because they think it will help their business.”

Meanwhile, the sales and marketing departments can’t get their heads around exactly what big data is for.

Is there a lack of workers skilled in data management?

Data management and analysis may have the power to unlock the potential of a business, but educated, professional big data experts are thin on the ground. “Most analysts in business are still enthusiastic early adopters, motivated to self-educate,” says Giles Slinger, Director at data analytics firm Concentra.

However, universities are now starting to create courses in analytics. In the UK, the Nuffield Foundation’s Q-Step programme is advancing analytics at 15 universities, while international business schools are now offering MBAs in big data and business analytics.

“These will generate graduates in one to three years,” says Slinger. “In the meantime, businesses have an opportunity to educate employees about the benefits of data analysis in supporting their daily tasks and integrating operations.”

Tom Pohlmann, Head of Values and Strategy at Mu Sigma

Finding the right people

It could be a long three years; recruitment website Indeed.com currently lists over 750,000 data analytics jobs, yet these positions are not being filled. “75% of companies are now actively recruiting people with skills in advanced data analytics,” says Tom Pohlmann, Head of Values and Strategy at Mu Sigma, “but it can be difficult to find candidates with the creativity and experimental mind-set to truly revolutionise the handling of big data to truly transform a business’ approach.”

The new trend for personalised advertising will put even more pressure on graduates, who’ll need a “scientific, artistic and emotive mind-set” if they’re to cope with the coming phase of extreme experimentation.

Black box approach

The ‘black box’ approach

If you want genuinely useful business insight, you have to work at it. “Too many think an analytics strategy means choosing a specific black box to be fed with specific data feeds, and applying specific pre-set algorithms,” says Nick Clarke, Head of Analytics at international analytics, software and consulting services company Tessella. “But many problems are too complex and too subtle to automate in this way … the correlations that pop out the other end are not magically wrapped up in valuable business insight.”

All big data is not the same, and not even Google can get insight from any unstructured data set. See the failure of the Google Flu Trends (GFT) project for proof of that – complex issues require sophistication when it comes to big data.

Getting it right

However, there are plenty of examples of getting it right. “The pharmaceuticals industry has long used vast complex data sets to identify profitable areas for new research,” says Clarke. “Although still far from perfect, their maturing combination of data analytics and clarity of vision of what they want to achieve, puts them at the front of the game.”

Their secret is the embedding of scientifically literate data experts within specialist research groups. “Human expertise is vital when framing the problem, contextualising the insight, and uncovering bias, both in the data feeds and in our assumptions,” says Clarke. “The data strategies of too many organisations fail at this level.”

Pure Storage's research suggests that a lack of accurate information can cost a business up to £20 million per year

On the cusp?

Some think we’re now on the cusp of being able to deliver insights on a computational scale. “The information age is finally finding its ‘steam engine’,” says Pohlmann. “This alignment of scale and technology means that we can now create algorithms to support things like machine learning processes, which sees big data evolve from being purely a numbers game to an integral part of any corporate planning process.”

Is big data a big failure so far?

“No,” says Pohlmann, who thinks that the problem is that many organisations are wedded to just their existing Enterprise Data Warehouse (EDW), usually in silos, and practicing business intelligence techniques that fall far short of the proactive, predictive operational analytics that can actually deliver on intelligent decision making.

“With the continuing rise of IoT, AI and machine learning, there’s going to be more pressure than ever applied to organisations that are stuck in the past to adopt new processes and techniques,” he says, “and actually start to benefit from what big data has to offer.”

The pace of change

Benefitting from big data can take time, but companies paying only lip service to big data are putting everything at risk. “As companies gather more and more granular data on what they do, the potential to gain understanding and plan accordingly is not just a profitable undertaking, it is a necessity,” says Petter.

“Transformation is being forced on organisations at an ever-increasing pace. They must adapt to new ways of doing business, new markets and new practices.” Or what? “Or die,” says Petter.

  • Also check out: Forget big data – it’s time for big algorithms to change the world