Redis as a Vector Database | RedisCloud

4 min readNov 8, 2022

An entry in a series of blogs written during the Vector Search Hackathon organized by the MLOps Community, Redis, and Saturn Cloud.

I’ve used Redis extensively before, but like most, I’ve only used it as a cache. So I was quite excited coming into the Vector Search Hackathon; it’s always fun when something you’re familiar with still holds secrets.

In this blog, we’ll look over different aspects of Redis and how we can use it as a vector database that’s capable of vector similarity search.

Managed Redis

As I noted here, I’m a fan of self-hosted software and that includes deploying Redis wherever we’d needed it previously. But working with RedisCloud now, it’s hard to understate how much simpler using high-availability Redis has become. I don’t have to think about sharding, availability, uptime, throughput, etc. This is perfect for teams that are unable to devote resources to maintaining a large, self-hosted Redis deployment.

Redis Modules

Redis is a key-value store. But there are certain tasks you can do on top of that store, and instead of having the user implement those externally, Redis Modules bring a lot of common use-cases like search, time-series data, and graphs into the Redis DB directly. For the hackathon, we primarily used Redisearch, which offers basic and full-text search and vector-similarity search. The API is very mature and the Python SDKs are great to work with.

Data Loading

I shouldn’t be surprised, but I was blown away by how quick Redis is at loading a ton of vectors into the Database. It’s very reasonable to do this near real-time. So if you have a use-case where an application needs vectors.

Redis Object Mapping Python

This is probably one of my favourite tools that I got from this already great experience. Using Pydantic models, we’re able to create detailed data models that can be directly stored/queried in/from Redis without ever having to look at a single backend command like hget.

class Paper(HashModel):
    paper_id: str = Field(index=True)
    title: str = Field(index=True, full_text_search=True)
    year: int = Field(index=True)
    authors: str
    categories: str
    abstract: str = Field(index=True, full_text_search=True)
    input: str

Since we’re dealing with Pydantic, there are no holds barred on validation — perfect!

Here, you’ll notice that the paper_id, title, year, and abstract variables are of type Field. The title and abstract alone though have full_text_search set to True. Now whenever equality or range checks are done on paper_id or year, it’ll use an index to speed things up. And when you search for specific words in the title or abstract, Redisearch will use full-text search to deliver the papers containing those words to you.

paper_instance = Paper(
    paper_id="0704.3780",
    title="Stochastic Optimization Algorithms",
    year="2006",
    categories="cs.NE",
    abstract="---",
    input="---",
)

This will now create an instance of the Paper class on your machine, but it still hasn’t been sent over to Redis. When you now do the following, it will create a hash in Redis and save this data.

paper_instance.save()

Querying

When querying, you don’t have to worry about the hashes, you can just reuse the same model you used to create the hash: Paper.

A few examples:

# Return all papers after 2004
Paper.find(Paper.year > 2004).all()# Return the first paper that contains the word "generative"
# in their abstract.
Paper.find(Paper.abstract % "generative").first()

This makes code so much more readable. This is a great fit for situations where you want the rigidity of a strongly typed DB with advanced validation features, but at the same time don’t want the overhead that comes with using a proper RDB.

Vectors

Vectors in the DB have to be bytes. So if you’re dealing with Numpy Arrays, you’ll have to convert them to bytes form first before storing them.

An interesting problem is that the datatype choice is up to you, so if you choose np.float32 to store a vector to the DB and retrieve it using np.fromBuffer with the data type as np.float64, you’re going to end up with a vector that’s half the size being returned. So use the same data type on both ends, when the vector enters and exits the DB.

np.array(vec, dtype=np.float32).tobytes()

Vector Indexes

There are hashes and there are keys beneath those hashes. We store vectors as values under a specific key. So when indexing the vectors, we need a way to group vectors of a kind together. A practical example is what we did with Top2Vec. We have three different kinds of vectors: word, document, and topic. And needed three indexes, one for each kind of vector. So we kept the key names for all three types the same, it was still “vector”. But when creating the index, we can pass a prefix option that will let us filter out a specific set of hashes whose “vector” fields will be indexed. So in our case, we had three functional indexes: “papers”, “words”, and “topics” and the hashes for each type began thus: paper_vector:, word_vector:, topic_vector.

redis_conn.ft(index_name).create_index(
    fields=fields,
    definition=IndexDefinition(
        prefix=[prefix],
        index_type=IndexType.HASH
    ),
)

For a detailed explanation of vector similarity in redis, visit this document.

Vector Search

The vector search queries are rather straightforward:

You define:

The number of results you want to be returned.
The field which contains the vector.
The bytes representation of the vector for which you want to find the k-nearest neighbours.

With the Python client for Redis, this is made even simpler, especially when using code from Tyler Hutcherson’s library.

query = search_index.vector_query(number_of_results=50)redis_conn.ft("papers").search(
    query,
    query_params={
        "vec_param": np.array(vec, dtype=np.float32).tobytes()
    },
)

Something to note is that the redis-py library doesn’t seem to be directly communicating with the API. Rather, it’s just constructing CLI commands through a Pythonic interface.

I see a lot of doors opening up for my team with vector capabilities from Redis. Just having it as a live vector store is more than valuable enough. We’re really looking forward to working further with Redisearch and try out more efficient ways of storing vectors and retrieving them.