By KEVIN POOL
In the 15 years I have been handling Big Data or the analytics around it, never has the excitement around these topics been so high. I find myself speaking in more conferences about both, handling smaller group CXO talks and reading a lot more about it in the press.
So what’s the biggest change over the years?
Big Data and analytics discussions are no longer limited to IT trade only. Marketing folks, business leaders and even product heads are eager to know more.
This has led to IT teams reviewing some Big Data themes with solid curiosity, and a few, who I interact with directly, have some R&D activities ongoing to evaluate Big Data technologies. A small percentage of the organisations have actually started implementing Big Data solutions to the point where they are getting meaningful business ROI.
Big Data successes
Review of the Big Data success stories, of which there are plenty in US but growing in the Far East, yields an interesting observation. The common factor in these success stories is the advanced capabilities to collect, integrate and process the data, then to gain insights and business value from the data.
Basically it’s not so much about the volume of the data, but an organisation’s ability to acquire, explore, understand, and leverage it.
While there are several success stories, there are still fewer than there ought to be.
Caution is in the air
Any discussion around Big Data brings caution among IT decision-makers, which at a level, is well founded. The technology, approaches and best practices around Big Data have been evolving rapidly over the previous years.
There are a large number of technologies and approaches that fall under the umbrella of Big Data, and the landscape continues to change rapidly. This leads people to question what is real and not hype, if now really is the right time to start initiatives, and if they will really get value from the initiatives.
As with all technology adoption cycles, there are been scary scenarios where some Big Data technology vendors, primarily funded by investor alone, as opposed to actual sales have vanished. Experienced CIOs therefore opine that Big Data is still in the hype cycle and not yet a mainstream solution.
IT decision-makers also ask, “What’s Different?” Some years back, Data Warehousing and Business Intelligence technologies and approaches were being pushed, which, at a level, sound similar in terms of concepts and advertised business benefits.
Surveys of organisations that invested in Data Warehouse initiatives show that many (if not most) of the organizations felt they invested significantly to put these items in place, yet have not received the promised business value. What was promoted was, “Build it and the business will invent ways to get value out of the information.” This messaging is strikingly similar to how Big Data solutions are being positioned.
Shock of much more effort required and then not discovering much
In practice it took much more effort than expected to get the data into the data stores. After the large amount of data finally made it into the data store, organizations found that the business or their data analysts did not find the earth-shaking business insights that had been advertised. This was due to several reasons.
The time and effort to create reports against the data store was larger than expected, and the cycle times inhibited the discovery process. Sometimes the analysts of the data were isolated into small business units, and widespread use and access to the data was not possible.
The end result was that there was a large effort expended to get the data into the data store, and very little business value derived from the data. The industry learned that to get insights and discovery requires a very interactive manipulation and exploration of the data, and the techniques were just too cumbersome for those objectives to be achieved.
All’s well that starts well — transformation of Big Data and Analytics
The good news is that technologies to address these issues have evolved substantially in recent years and are now lot more robust and interoperable with existing IT infrastructures. Capabilities of these solutions have kept pace with the increase in data volumes.
Application Integration has matured to the point where it is now a relatively routine task to integrate complex, high-volume data sources. Brilliant log management technologies that can collect information from many systems, across geographies, are also available — an example being Procter & Gamble, with over 4.8 billion customers in over 180 countries using advanced integration capabilities to collect and process the diverse information and incorporate it into active Big Data analysis for spotting opportunities.
Data cleansing, data translation, and master data management tools and techniques have also matured to the point that it is feasible to process complex, dirty data sets with much greater confidence than earlier years.
Using these technologies, companies in the USA like Macy’s and Kroger are managing customer and product information and are incorporating this information in real-time Big Data inventory and customer management processes.
The concerns about getting good, clean data into the data store can now be comfortably addressed. These areas have even matured to the point that there are cloud-based offerings for data integration, data cleansing and master data management, significantly lowering the cost and long lead times.
For example, TIBCO now has cloud based offerings for integration, data cleansing, master data management, grid computing and visual analytics.
Analytics incisive but also well presented
As I had mentioned earlier, dramatic improvements in analytics have started building the group of converts. Data presented in a tedious manner automatically kills interest in people, along with productivity and the desire to go deeper.
Not only are the analytics tools and techniques much stronger now, they appear much more mainstream in the areas of interactive visual analytics, are able to provide powerful statistical modeling and predictive analytics. Naturally, data users are attracted to spending less time and getting much more out of Big Data.
With these tools, both data analysts and businesses can interactively explore and use data to gain new insights. Data visualization, better display of Big Data and data discovery tools, for example, can help banks in determining the main cause of the dissatisfaction among a high percentage of customers.
They can link and find relationships between a rich set of interaction points and customer satisfaction scores. For example, a moderate sized EU Telco was able to use this information to create customer retention promotions which resulted in US$40mil (RM129.6mil)/Year of increased revenue.
It’s now possible to publish focused data subsets for audiences throughout the enterprise, which gives users capabilities of interacting with the data, and gaining the insights and discovery tailored to their specific needs.
This takes the Big Data out of a small Business Intelligence team and expands its use and benefits throughout the enterprise.
At the end of the day, there are proven technologies and newer technologies available which can usher in business benefits, but it is the leaders in the organisation who will have to bring about the change required to adopt and adapt to get the Big Value from their data initiatives.
(Kevin Pool is chief technology officer at TIBCO Asia.)