Personal project
Marconi
Redis from scratch: an in-memory data server in Go with RESP, TTL expiry, an append-only file and PSYNC replication.
redis-cli
> SET book byor
OK
> GET book
"byor"
> HSET user name elhadji
(integer) 1
> RPUSH q a b c
(integer) 3
> EXPIRE book 60
(integer) 1
> TTL book
(integer) 59
# server restarted; the AOF replays
> HGETALL user
1) "name"
2) "elhadji"
Problem
Redis looks simple from the outside, but the hard parts are hidden: how a protocol is parsed from a stream, how keys expire without slowing every read, and how durability and replication work. Building a smaller version was the fastest way to learn them.
What I built
Marconi speaks RESP, so it works with a Redis-style client. It stores strings, hashes, lists and sets, supports EXPIRE and TTL, persists writes to an append-only file, and can sync a read-only replica.
It ships with a small CLI and a web key browser backed by a JSON API.
How
- TCP connectionsOne RESP reader and writer per client
- Command dispatchGET, SET, HSET, LPUSH, SADD, EXPIRE and more
- StoreHashtable and expiry heapAppend-only fileAppend, replay, rewrite
- ReplicationPSYNC, empty RDB frame, snapshot as commands, live writes
Expiry sits on a heap, so the server can find the next key to expire without scanning the keyspace. Writes are appended to the AOF and replayed on start, and a rewrite compacts the log.
A replica sync runs PSYNC, receives an empty RDB frame and a snapshot of current keys as commands, then follows live writes.
Results
- Works with a RESP client across four data types, including TTL expiry.
- State survives a restart through AOF replay, as shown in the session above.
- The limits are explicit: one database, no sorted sets, streams, pub/sub, transactions, Lua, ACLs or eviction. It is not a drop-in Redis.
What I learned
- Durability is a design decision that touches every write path, not something to add at the end.
- Keeping the first replication handshake small made the moving parts easier to reason about.