In-memory computing, and variations of in-memory databases, have been around for some time. But only in the last couple of years has the technology advanced and the cost of memory declined enough that in-memory computing has become cost effective for many enterprises. Major research firms like Gartner have taken notice and have started to focus on broadly applicable use cases for in-memory databases, such as Hybrid Transactional/Analytical Processing (HTAP for short).
HTAP represents a new and unique way of architecting data pipelines. In this chapter we will explore how in-memory database solutions can improve operational and analytic computing through HTAP, and what use cases may be best suited to that architecture.
There are two primary categories of database workloads that can suffer from delayed access to data. In-memory databases can help in both cases.
OLTP workloads are characterized by a high volume of low-latency operations that touch relatively few records. OLTP performance is bottlenecked by random data access—how quickly the system finds a given record and performs the desired operation. Conventional databases can capture moderate transaction levels, but trying to query the data simultaneously is nearly impossible. That has led to a range of separate systems focusing on analytics more than transactions. These online analytical processing (OLAP) solutions complement OLTP solutions.
However, in-memory solutions can increase OLTP transactional throughput; each transaction—including the mechanisms to persist the data—is accepted and acknowledged faster than a disk-based solution. This speed enables OLTP and OLAP systems to converge in a hybrid, or HTAP, system.
When building real-time applications, being able to quickly store more data in-memory sets a foundation for unique digital experiences such as a faster and more personalized mobile application, or a richer set of data for business intelligence.
OLAP becomes the system for analysis and exploration, keeping the OLTP system focused on capture of transactions. Similar to OLTP, users also seek speed of processing and typically focus on two metrics:
Traditionally, OLAP has not been associated with operational workloads. The “online” in OLAP refers to interactive query speed, meaning an analyst can send a query to the database and it returns in some reasonable amount of time (as opposed to a long-running “job” that may take hours or days to complete). However, many modern applications rely on real-time analytics for things like personalization and traditional OLAP systems have been unable to meet this need. Addressing this kind of application requires rethinking expectations of analytical data processing systems. In-memory analytical engines deliver the speed, low latency, and throughput needed for real-time insight.
When working with transactions and analytics independently, many challenges have already been solved. For example, if you want to focus on just transactions, or just analytics, there are many existing database and data warehouse solutions:
However, rapidly emerging workloads are no longer served by any of the traditional options, which is where new HTAP-optimized architectures provide a highly desirable solution. HTAP represents a combination of low data latency and low query latency, and is delivered via an in-memory database. Reducing both latency variables with a single solution enables new applications and real-time data pipelines across industries.
Near ubiquitous Internet connectivity now drives modern workloads and a corresponding set of unique requirements. Database systems must have the following characteristics:
HTAP-capable systems can run analytics over changing data, meeting the needs of these emerging modern workloads. With reduced data latency, and reduced query latency, these systems provide predictable performance and horizontal scalability.
In-memory databases deliver more transactions and lower latencies for predictable service level agreements or SLAs. Disk-based systems simply cannot achieve the same level of predictability. For example, if a disk-based storage system gets overwhelmed, performance can screech to a halt, wreaking havoc on application workloads.
In-memory databases also deliver analytics as data is written, essentially bypassing a batched extract, transform, load (ETL) process. As analytics develop across real-time and historical data, in-memory databases can extend to columnar formats that run on top of higher capacity disks or flash SSDs for retaining larger datasets.
Applications driving use cases for HTAP and in-memory databases range across industries. Here are a few examples.
Agile businesses need to implement tight operational feedback loops so decision makers can refine strategies quickly. In-memory databases support rapid iteration by removing conventional database bottlenecks like disk latency and CPU contention. Analysts appreciate the ability to get immediate data access with preferred analysis and visualization tools.
Successful companies must be able to quantify and plan for risk. Risk calculations require aggregating data from many sources, and companies need the ability to calculate present risk while also running ad hoc future planning scenarios.
In-memory solutions calculate volatile metrics frequently for more granular risk assessment and can ingest millions of records per second without blocking analytical queries. These solutions also serve the results of risk calculations to hundreds of thousands of concurrent users.
Today’s users expect tailored experiences and publishers, advertisers, and retailers can drive engagement by targeting recommendations based on users’ history and demographic information. Personalization shapes the modern web experience. Building applications to deliver these experiences requires a real-time database to perform segmentation and attribution at scale.
In-memory architectures scale to support large audiences, converge a system or record with a system of insight for tighter feedback loops, and eliminate costly pre-computation with the ability to capture and analyze data in real time.
Financial assets and their value change in real time, and the reporting dashboards and tools must similarly keep up. HTAP and in-memory systems converge transactional and analytical processing so portfolio value computations are accurate to the last trade.
Now users can update reports more frequently to recognize and capitalize on short-term trends, provide a real-time serving layer to thousands of analysts, and view real-time and historical data through a single interface (Figure 1-1).
The increase in connected applications drove a shift from logging and log analysis to real-time event processing. This provides businesses the ability to instantly respond to events, rather than after the fact, in cases such as data center management and fraud detection. In-memory databases ingest data and run queries simultaneously, provide analytics on real-time and historical data in a single view, and provide the persistence for real-time data pipelines with Apache Kafka and Spark (Figure 1-2).
In the early days of databases, systems were designed to focus on each individual transaction and treat it as an atomic unit (for example, the debit and credit for accounting, the movement of physical inventory, or the addition of a new employee to payroll). These critical transactions move the business forward and remain a cornerstone of systems-of-record.
Yet, a new model is emerging where the aggregate of all the transactions becomes critical to understanding the shape of the business (for example, the behavior of millions of users across a mobile phone application, the input from sensor arrays in Internet of Things (IoT) applications, or the clicks measured on a popular website). These modern workloads represent a new era of transactions requiring in-memory databases to keep up with the volume of real-time data and the interest to understand that data in real time.