State of the Hybrid Data Ecosystem in 2015
For each of the past three years, Enterprise Management Associates (EMA) has conducted a primary research study directed at the evolution of data management platforms associated with Big Data. These studies have shown that data management platforms supporting Big Data initiatives cannot be pigeon-holed into a technical constraint such as “just” Hadoop… or “just” NoSQL… or “just” historical analytics…or just the data center. Organizations around the world are using a wide range options that are driven more by the business requirements of their Big Data initiatives than by the technical limitations of their data management platforms. Organizations focus more on finding the correct tool for the job than attempting to fit a square data element into a round table.
For example, many organizations leverage the economical storage and processing attributes of Hadoop in association with the speed of response of an operational platform (NewSQL or NoSQL) or an analytical database. In these cases, data is stored and prepared where it makes the most sense. A subset of that information is then shared with a higher performance platform to meet the workload associated with a given use case.
For example use cases can be the operational business processes that maintain a given organization’s online or mobile presence. Use cases can also be the near-real-time distribution of core analytical results. Finally, they can be the integration of analytics directly into one or more of those operational processes in a technique that EMA refers to as operational analytics, such as real-time evaluation for fraud management in customer retail settings or risk management in financial settings where external regulatory or internal exposure management controls need to be honored.
In 2013, the EMA/9sight study identified the diversity of these platforms including operational data stores including NewSQL; NoSQL platforms; external data sources; Hadoop; analytical appliances; enterprise data warehouses; data marts; and discovery platforms. In 2014, the study examined the nature of the relationships between these platforms in end-user technical environments. Nearly 65% of organizations were using between two and four of those platforms to accomplish their various project goals. In 2015, the research focused on the various implementation avenues for these data management platforms and the degree of latency for these various platforms.
In terms of how various data management platforms are implemented, the 2015 EMA/9sight study determined that while organizations are still using bare metal implementations in their data centers, they are branching out into cloud-based implementations in general with private cloud. In particular, companies are venturing outside their data centers into public cloud resources, hybrid environments with shared resources inside and outside the data center and managed service implementations where the management and operation of data management platforms serving Big Data are handled by a 3rd party.
The chart above shows that enterprise data warehouses have the slowest adoption in general in terms of cloud. However, a significant number of end-users are still utilizing non-traditional implementation strategies. The most cloud-implemented data management platform is External Data Sources. Often these data sources are managed by government agencies or data aggregators and provide the contextual information required to meet a wide range of workloads from operational to exploratory.
The research shows changes in importance of speed of response to end-users from last year to this year. This year EMA/9sight respondents were asked to provide information on the types of latency required by their workloads.
Near-real-time dominated the project workloads with an emphasis on the operational and operational analytical workloads. “Purely” analytical and exploratory workloads had much lower time-sensitive processing requirements. The result of this research is the newly-updated EMA Hybrid Data Ecosystem (HDE).
The HDE continues to provide organizations with a guide relating to their implementations and strategies associated with Big Data not from a strictly technically perspective, but rather how the requirements of the business are and should be driving the operational and analytical implementations of Big Data.
Managing Research Director, Enterprise Management Associates, has years of experience working in areas related to business analytics in professional services consulting and product development roles. He has also helped organizations solve their business analytics problems whether they relate to operational platforms, such as customer care or billing, or applied analytical applications, such as revenue assurance or fraud management.