Spring Boot with Redis: Pipeline Operations

Introduction

Redis is an in-memory data store, which can be used as a NoSQL database, cache, or as a typical message broker. It's written in ANSI C, which compiles into significantly efficient machine code and its ability to store data as key-value pairs makes in-memory caching an attractive use-case for Redis, besides also persisting data to a disk.

In this article, we'll use Pipelining to allow a Spring Boot application to send multiple requests to the Redis server, in a non-blocking way.

Use Case of Pipelining in Redis

Redis is based on a Client/Server(Request/Response) architecture. In these architectures, a client typically sends a query or request to the server and waits for a response. This is usually done in a blocking way such that a new request can't be sent until the response for the last one was sent:

Client: <command 1>
Server: Response for <command 1>
Client: <command 2>
Server: Response for <command 2>
Client: <command 3>
Server: Response for <command 3>

This can lead to massive inefficiencies, with high latency between receiving commands and processing them.

Pipelining allows us to send multiple commands, as a single operation by the client, without waiting for the response from the server between each command. Then, all the responses are read together instead:

Client: <command 1>
Client: <command 2>
Client: <command 3>
Server: Response for <command 1>
Server: Response for <command 2>
Server: Response for <command 3>

Since the client is not waiting for the server's response before issuing another command, latency is reduced which in turn improves the performance of the application.

Note: The commands here are placed in a queue. This queue should remain of a reasonable size. If you're dealing with tens of thousands of commands, it's better to send and process them in batches, lest the benefits of pipelining become redundant.

Implementation

Let's go ahead and make a small Spring Boot application that works with Redis and pipelines multiple commands. This is made easier with the help of the Spring Data Redis project.

Spring Boot Setup

The easiest way to start off with a blank Spring Boot app is to use Spring Initializr:

Alternatively, you can also use the Spring Boot CLI to bootstrap the application:

$ spring init --dependencies=spring-boot-starter-data-redis redis-spring-boot-demo

We're starting off with the spring-boot-starter-data-redis dependency as it includes spring-data-redis, spring-boot-starter and lettuce-core.

If you already have a Maven/Spring application, add the dependency to your pom.xml file:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
    <version>${version}</version>
</dependency>

Or if you're using Gradle:

compile group: 'org.springframework.data', name: 'spring-data-redis', version: '${version}'

We'll also be using Jedis as the connection client, instead of Lettuce:

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
    <version>${version}</version>
</dependency>

Redis Configuration

We will host Redis on ScaleGrid, which provides a free trial account, for hosting a Redis server instance. Alternatively, you can download the server and host it on your own computer on Linux and MacOS. Windows requires a bit of hacking and is tricky to set up.

Let's set up the JedisConnectionFactory so that our application can connect to the Redis server instance. In your @Configuration class, annotate the adequate @Bean:

@Configuration
public class Config {
    @Bean
    public JedisConnectionFactory redisConnectionFactory() {
        JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory();
        jedisConnectionFactory.setHostName("<server-hostname-here>");
        jedisConnectionFactory.setPort(6379);
        jedisConnectionFactory.setPassword("<server-password-here>");
        jedisConnectionFactory.afterPropertiesSet();
        return jedisConnectionFactory;
    }
}

RedisTemplate is an entry-class provided by Spring Data through which we interact with the Redis server.

We'll pass our configured connection factory to it:

@Bean
public RedisTemplate<String, String> redisTemplate() {
    RedisTemplate<String, String> redisTemplate = new RedisTemplate<>();
    redisTemplate.setConnectionFactory(redisConnectionFactory());
    redisTemplate.setDefaultSerializer(RedisSerializer.string());
    redisTemplate.afterPropertiesSet();
    return redisTemplate;
}

We have set the default serializer to store keys and values as String. Alternatively, you can define your own custom serializer.

Pipelining Using RedisTemplate

Let's add a few items from a list to the Redis server. We'll do this without pipelining, using the RedisTemplate. Specifically, we'll use the ListOperations interface, acquired from opsForList():

List<String> values = Arrays.asList("value-1", "value-2", "value-3", "value-4", "value-5");
redisTemplate.opsForList().leftPushAll("pipeline-list", values);

Running this code will result in:

Now, let's remove these. Imagining that this can be an expensive operation, we'll pipeline each rPop() command so that they get sent together and that the results synchronize upon all of the elements being removed. Then, we'll receive these results back. To pipeline commands, we use the executedPipeline() method.

It accepts a RedisCallback or SessionCallback that we provide it with. The executedPipeline() method returns the results that we can then capture and review. If this isn't needed, and you'd just like to execute the commands, you can use the execute() method, and pass true as the pipeline argument instead:

List<Object> results = redisTemplate.executePipelined(new RedisCallback<Object>() {
    public Object doInRedis(RedisConnection connection) throws DataAccessException {
        for(int i = 0; i < 5; i++) {
            connection.rPop("pipeline-list".getBytes());
        }
    return null;
     }
});
return results;

The executePipelined() method accepts a new RedisCallback(), in which we use the doInRedis() method to specify what we'd like to do.

Specifically, we've run the rPop() method 5 times, removing the 5 list elements we've inserted beforehand.

Once all five of these commands have been executed, the elements removed from the list and sent back - the results are packed in the results list:

Conclusion

Redis' most popular use-case is as a cache store. However, it can also be used as a database, or as a message broker.

Redis allows us to increase the performance of applications, by minimizing the calls to the database layer. Its support for pipelining allows for multiple commands to be sent to the server in a single write operation, thereby reducing round trip Time to and from the server.

In this article, we've pipelined multiple commands using the RedisTemplate API and verified the results.