Build your own Redis from scratch, and talk to it with the real redis-cli

Redis has a reputation for being a serious piece of infrastructure, and it is. But the core of it, the part that makes it Redis, is astonishingly small. Small enough that you can rebuild it in about 80 lines of Python, point the real redis-cli at your version, and have it just work. Same commands, same wire protocol, same behavior.

That is the fun of it. By the end you will run redis-cli -p 6399 set foo bar, and the OK that comes back is from a server you wrote. This is the written companion to the video build, and it goes one step further at the end: how an in-memory database survives being turned off and on again.

The one idea

Redis is a dictionary you talk to over a network socket. That is genuinely the whole concept. The data lives in memory, which is why it is fast, and you reach it with a protocol so simple you can read it by eye. Everything else, the data types, persistence, replication, is built on top of that one move.

RESP, the wire protocol

A command sent to Redis is just an array of text. When you type SET foo bar, the client sends this over the socket:

*3
$3
SET
$3
foo
$3
bar

*3 means "three arguments are coming." Each argument is $<length> followed by the value. That is RESP, the REdis Serialization Protocol. It is human-readable and trivial to parse, and that simplicity is a feature, not a limitation.

Replies use the same tagging idea: the first byte tells you the type. + is a simple string, - an error, : an integer, $ a bulk string, and $-1 is the null reply. So we write one tiny function per reply type:

def simple(s):  return f"+{s}\r\n".encode()
def error(s):   return f"-ERR {s}\r\n".encode()
def integer(n): return f":{n}\r\n".encode()
def bulk(s):
    if s is None:
        return b"$-1\r\n"
    b = s.encode()
    return f"${len(b)}\r\n".encode() + b + b"\r\n"

Reading a command is the mirror image. The first line tells us how many arguments to expect; for each one we skip the length line and take the value:

def read_command(f):
    line = f.readline()
    if not line:
        return None
    n = int(line[1:])            # "*3\r\n" -> 3
    args = []
    for _ in range(n):
        f.readline()             # "$3\r\n", the length, which we do not need
        args.append(f.readline().rstrip(b"\r\n").decode())
    return args

The commands

Now the part that feels like Redis: a big if-else on the first word. The store is a plain Python dict.

store = {}

def handle(args):
    cmd = args[0].upper()
    if cmd == "PING":  return simple("PONG")
    if cmd == "SET":
        store[args[1]] = args[2]
        return simple("OK")
    if cmd == "GET":   return bulk(store[args[1]] if alive(args[1]) else None)
    if cmd == "DEL":   return integer(sum(1 for k in args[1:] if store.pop(k, None) is not None))
    if cmd == "INCR":
        v = int(store.get(args[1], 0)) + 1
        store[args[1]] = str(v)
        return integer(v)
    ...

A few things are quietly true here. The store really is just a dictionary; Redis is, at heart, one in-memory hash map. Numbers are not a separate type, INCR reads a string, does math, and writes a string back. And GET on a missing key returns the null reply, which is why redis-cli prints (nil).

Expiry, and the clever bit

Keys can expire. EXPIRE stamps a key with a death time, and TTL reports the seconds left:

expiry = {}    # key -> unix timestamp when it dies

def alive(key):
    if key in expiry and time.time() > expiry[key]:
        store.pop(key, None)
        expiry.pop(key, None)
    return key in store

Notice what is missing: a timer. We never run a background job scanning for dead keys. A key dies the moment you reach for it and we notice it is overdue. That is lazy expiration, and real Redis does exactly this, plus a little random background sweeping so memory from never-touched expired keys eventually gets reclaimed.

The server

The whole server is two short functions: serve one client by looping over its commands, and accept new clients into their own threads.

def serve_client(conn):
    f = conn.makefile("rb")
    while True:
        args = read_command(f)
        if not args:
            break
        conn.sendall(handle(args))
    conn.close()

def serve(port=6399):
    s = socket.socket()
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(("127.0.0.1", port))
    s.listen()
    while True:
        conn, _ = s.accept()
        threading.Thread(target=serve_client, args=(conn,), daemon=True).start()

Run it, and in another terminal:

$ redis-cli -p 6399 set name iwtlp
OK
$ redis-cli -p 6399 get name
"iwtlp"
$ redis-cli -p 6399 incr views
(integer) 1

That is the actual redis-cli, talking to our 80-line server, because we speak its protocol. We did not mimic Redis. For these commands, we are Redis.

How Redis survives a restart

Here is the question the video only teases, and the most interesting one: if the entire database lives in RAM, what happens when the process dies? Everything is gone. Real Redis solves this two ways, and both are simple enough to build.

RDB: snapshots

The first approach is a point-in-time snapshot. Every so often, dump the whole dataset to a file. On startup, load it back.

import json

def save_rdb(path="dump.rdb"):
    json.dump({"store": store, "expiry": expiry}, open(path, "w"))

def load_rdb(path="dump.rdb"):
    if os.path.exists(path):
        data = json.load(open(path))
        store.update(data["store"])
        expiry.update(data["expiry"])

Real Redis uses a compact binary format and forks a child process so the snapshot does not block the server, but the idea is exactly this: freeze the dict, write it down. Snapshots are small and load fast, which makes restarts quick. The cost: you lose every write made since the last snapshot. If you snapshot every five minutes and crash at minute four, those four minutes are gone.

AOF: the append-only log

The second approach trades the snapshot for a journal. Every write command is appended to a log file as it happens. On startup, you replay the log and the state rebuilds itself.

aof = open("appendonly.aof", "a")
WRITES = {"SET", "DEL", "INCR", "EXPIRE"}

def log_write(args):
    if args[0].upper() in WRITES:
        aof.write(" ".join(args) + "\n")
        aof.flush()

def replay_aof(path="appendonly.aof"):
    if os.path.exists(path):
        for line in open(path):
            handle(line.split())     # re-run every past write, in order

The key insight: you only log writes, never reads, and replaying them in order reconstructs the exact state. This is the same idea as a database write-ahead log, or, if you watched the git build, the same spirit as replaying commits. AOF is far more durable, you can lose at most one write, but the log grows forever, so real Redis periodically rewrites it into the shortest set of commands that produces the current state.

The two are not exclusive. Modern Redis can run both: snapshots for fast restarts, the append log for durability, and on boot it prefers the log because it is more complete. Persistence, the thing that sounds like deep magic, is just "write the dict down" or "write down every change."

What is left

Real Redis adds replication so a follower can take over, pub/sub for messaging, and the rich data types people actually reach for: lists, sets, hashes, and sorted sets, each a small structure layered over the same key space. It also runs single-threaded through an event loop instead of a thread per client, which is why it never needs locks. Those are real engineering, but the spine you just built, a dict behind a dead-simple protocol, is what all of it stands on.

If you want to build the rest, the full from-scratch track is on iwtlp.com. Learn it by building it.

推荐订阅源

DEV Community