Best practices in big data analytics, business intelligence and reporting
With the rise of big data, the database and data management tools market is in a state of flux, the likes of which have not been seen in this sector before. Companies are now awash in big data, and end users are demanding greater capability and integration to mine and analyze new sources of information. As a result, organizations are supplementing their relational database environments with new platforms and approaches that address the variety and volume of information being handled.
Download this Database Trends and Applications best practices report to learn more about how to get value out of the all the data that your organization is currently capturing with smarter business intelligence and data analytics.
We've also pulled out the first several pages of the whitepaper for you to read. Download the PDF on the right to read the rest.
Big Data Opens New Frontiers for the Reporting and Analytics Industry
With the rise of big data, the database and data management tools market is in a state of flux, the likes of which have not been seen in this sector before. Companies are now awash in big data, and end users are demanding greater capability and integration to mine and analyze new sources of information. As a result, organizations are supplementing their relational database environments with new platforms and approaches that address the variety and volume of information being handled.
Notably, big data is fueling the emergence of a new generation of database types, which no longer sit at the fringes of enterprises. While predictions of the demise of relational databases have abounded for more than a decade, a new survey of 264 data and IT managers finds that big data is adding a new wrinkle to this speculation. The survey, conducted among DBTA readers by Unisphere Research, the market research unit of Information Today, Inc. and sponsored by MarkLogic, confirms that these databases are not only being implemented for peripheral applications but also for core functions. The most prevalent platforms seen are the so-called NoSQL (“not only” SQL) and in-memory databases. Adoption of these nonrelational databases is on the rise as big data expands. However, many respondents worry about the skill sets they will need to effectively deliver solutions on these platforms.
The value of being able to manage big data is not lost on major vendors, and there has been a spate of acquisitions in recent years to fill any gaps that exist in product lines. For example, IBM recently entered into an agreement to buy Vivisimo, creator of big data analysis software. VMware bought Vetas, a big data analytics startup; Teradata purchased Aster Data Systems, an unstructured data analysis provider; HP purchased Vertica; IBM acquired Netezza; and EMC bought Greenplum. There have also been a number of strategic partnerships forged, including IBM’s partnership with Cloudera, a Hadoop provider. Cloudera is also partnering with both Oracle and Teradata. In addition, large, established data players have entered the big data domain with their own offerings, Vish Vishwanath, senior vice president of BI and analytics at Persistent Systems, relates to DBTA. “These have included the integration of big data with traditional BI and reporting solutions, appliance-based solutions that bundle big data technology with other software product offerings on a hardware platform, or as cloud-based big data solutions.”
Ultimately, businesses understand the intrinsic value in the large data stores they are accumulating. “Big data should be thought of as a new natural resource,” Deepak Advani, vice president of IBM Business Analytics Products and SPSS, tells DBTA. “Data and analytics are abundant resources and companies are seeing how they can drive those into new solutions—in particular, retaining and satisfying customers and increasing operational efficiency.”
In addition, big data affords companies the opportunity to ask questions they never were able to ask before, adds Christian Hasker, director of product development for Quest Software. “How can they bring big data techniques to their current data and gain that competitive advantage? How can financial organizations find better ways to detect fraud? How can insurance companies gain a deeper insight into their customers to see who may be the most un-economical to insure? How does a software company find their most at-risk customers—those who are about to deploy a competitive product? These and countless other questions can be answered by adopting big data techniques.”
But amid all the hype and excitement around big data, it’s important to keep things in perspective as well. Big data itself may also carry some erroneous assumptions that data managers need to understand and convey to their businesses. “The manipulation of the data into truly holistic data points on top of which to base cogent decisions, is far harder than it looks,” Antonio Piraino, CTO of ScienceLogic, tells DBTA. “There is a ways to go before the variety of correlations and presentations of big data are refined and proven. It’s not always easy to confidently interpret what the data is telling you; not all values captured from big data are immediately relevant to the symptom of a problem or imminent opportunity.”
The rise of new and bigger data environments also brings the risk of lack of cohesion within enterprises, many of which are still dominated by traditional data environments. “The big database vendors all follow the same general architecture,” noted Brian Clark, vice president of product management for Objectivity, Inc., in a webcast the company presented in partnership with DBTA. “Capture of the data coming from many sources at high speed, do some initial processing, then put it into some other storage for further processing and analysis, and then use that data in data analytics or visualization applications. The problem with this approach is yes, they can get the volume of data in, and yes, they can get the variety. But the data still ends up in different stores or different silos. And we believe there are hidden relationships in the data in those silos.”
In many organizations, “traditional data sources are not well integrated,” agrees Bill Abbott, principal for PwC, noting that “80% to 90% of the productivity cycle is spent gathering and organizing information.” Abbott recommends that organizations adopt a defined analytics strategy, focusing on the repeatability of valuable analytical processes.
Often, organizations attempting to manage big data have diverse environments in which it is difficult to actually take advantage of the data. “One of the problems of distributed big data is knowing what you’ve got and where it is,” says Clark, who recommends a federated approach to leverage data across the enterprise. “If you look at the ways big data is being handled, being managed, then typically it’s a combination of different storage, different databases, that store structured data.”