amintalebi · Oct 14, 2026 · Oct 14, 2026 · Oct 14, 2026 · Oct 14, 2026 · coderabbitai
diff --git a/.env.example b/.env.example
@@ -13,3 +13,11 @@ BUNDLER_VERSION=dev
 # ERC4337_BUNDLER_SUPPORTED_ENTRY_POINTS=0x433709009B8330FDa32311DF1C2AFA402eD8D009
 # ERC4337_BUNDLER_IS_OP_STACK_NETWORK=true
 # ERC4337_BUNDLER_IS_ARB_STACK_NETWORK=true
+
+# Optional: pluggable storage. By default the mempool and reputation data live
+# in a node-local BadgerDB under ERC4337_BUNDLER_DATA_DIRECTORY. To share state
+# across replicas, point the bundler at Redis/Valkey (see docs/STORAGE.md).
+# ERC4337_BUNDLER_STORE_BACKEND=redis
+# ERC4337_BUNDLER_STORE_URL=rediss://:REDIS_PASSWORD@redis.example.com:6379/0
+# ERC4337_BUNDLER_STORE_NAMESPACE=qubenode
+# ERC4337_BUNDLER_STORE_OP_TTL_SECONDS=180
diff --git a/CLOUD_NATIVE_ROADMAP.md b/CLOUD_NATIVE_ROADMAP.md
@@ -36,65 +36,75 @@ criteria.
 
 - [ ] No two bundler replicas may build/submit overlapping bundles for the same
       `(entryPoint, sender, nonce)`.
-- [ ] The mempool must survive a rolling deploy (no lost pending `UserOperation`s).
+- [x] The mempool must survive a rolling deploy (no lost pending `UserOperation`s).
 - [ ] Signer address used at startup must be verified and exposed as the beneficiary.
-- [ ] A missing/failed dependency must fail readiness, not liveness.
+- [x] A missing/failed dependency must fail readiness, not liveness.
 
 ---
 
 ## 1. Pluggable Storage
 
+> **Status:** the storage abstraction and backends are implemented in
+> `pkg/store`, documented in `docs/STORAGE.md`. The distributed locking/op-claim
+> wiring into the batch loop (1.4) is the remaining work; until then the
+> single-active-bundler topology is the supported shared-store deployment.
+
 ### 1.1 Define a storage interface
 
-- [ ] Introduce `pkg/store` (or `internal/store`) with narrow interfaces per concern:
-  - [ ] `MempoolStore` — `Add(ep, op)`, `Remove(ep, ops...)`, `Ops(ep, sender)`,
+- [x] Introduce `pkg/store` (or `internal/store`) with narrow interfaces per concern:
+  - [x] `MempoolStore` — `Add(ep, op)`, `Remove(ep, ops...)`, `Ops(ep, sender)`,
         `All(ep)`, `Clear()`, plus a way to stream by entity/nonce ordering.
-  - [ ] `ReputationStore` — `GetStatus`, `OpsCount`, `Increment*`, `Override`, `Dump`, `Clear`.
-  - [ ] `CodeHashStore` — `Save`, `Get`, `Remove`.
-  - [ ] `Closer` / `Pinger` — `Close()`, `Ping(ctx)`.
-- [ ] Return typed errors so callers/health can distinguish "not found" from
-      "backend unavailable" (mapping from `badger.ErrKeyNotFound`).
-- [ ] Keep interfaces context-aware (`ctx context.Context`) to support network
+  - [x] `ReputationStore` — `Status`, `OpsCount`, `Apply` (atomic signed deltas),
+        `Override`, `Dump`, `Clear`.
+  - [x] `CodeHashStore` — `Save`, `Get`, `Remove`, `Clear`.
+  - [x] `Store.Ping(ctx)` / `Store.Close()`.
+- [x] Return typed errors so callers/health can distinguish "not found" from
+      "backend unavailable" (`store.ErrNotFound`, `store.ErrUnavailable`).
+- [x] Keep interfaces context-aware (`ctx context.Context`) to support network
       backends and cancellation.
-- [ ] Add a `Factory`/`Open(cfg)` selector: `badger`, `redis`, `valkey`, `memory`.
+- [x] Add a `Factory`/`Open(cfg)` selector: `badger`, `redis`, `valkey`, `memory`
+      (`pkg/store/factory`).
 
 ### 1.2 BadgerDB backend (extraction, not removal)
 
-- [ ] Move existing Badger logic behind the new interfaces
-      (`pkg/mempool/instance.go`, `pkg/mempool/db.go`,
-      `pkg/modules/entities/reputation.go`, `pkg/modules/checks/db.go`).
-- [ ] Keep `badger` as the default for local dev / single-node deployments.
-- [ ] Remove the `*badger.DB` types from `internal/start/*`, `internal/start/health.go`,
+- [x] Move existing Badger logic behind the new interfaces
+      (`pkg/mempool/*`, `pkg/modules/entities/reputation.go`,
+      `pkg/modules/checks/db.go`); `pkg/mempool` is removed and all callers use
+      `store.MempoolStore`.
+- [x] Keep `badger` as the default for local dev / single-node deployments.
+- [x] Remove the `*badger.DB` types from `internal/start/*`, `internal/start/health.go`,
       and `internal/start/db.go`; inject the interface instead.
 
 ### 1.3 Redis/Valkey backend
 
-- [ ] Add a client dependency (`github.com/redis/go-redis/v9`; Valkey is wire-compatible).
-- [ ] Key layout (namespaced by deployment + chain ID):
-  - [ ] Mempool op: `mempool:{ep}:{sender}:{nonce}` → serialized `UserOperation` JSON.
-  - [ ] Arrival ordering: `ZSET mempool:order:{ep}` scored by arrival time/sequence.
-  - [ ] Per-sender index: `ZSET mempool:sender:{ep}:{sender}` scored by nonce.
-  - [ ] Per-factory / per-paymaster indexes: `ZSET mempool:entity:{ep}:{addr}`.
-  - [ ] Reputation counters: `HASH reputation:{ep}:{addr}` (opsSeen, opsIncluded).
-  - [ ] Code hashes: `mempool:codeHashes:{userOpHash}`.
-- [ ] Make writes atomic per op using `MULTI`/Lua so an op and all its indexes are
-      updated together (replacement of a same-`(sender,nonce)` op must not leak
-      stale index entries). Mirror `RemoveOps` cleanup at `pkg/mempool/queue.go:80`.
-- [ ] Apply a TTL to pending ops (`ERC4337_BUNDLER_MAX_OP_TTL_SECONDS`) so a crashed
-      replica cannot leave op state forever; reconcile TTL with `expire.DropExpired`.
-- [ ] Use a distributed lock / compare-and-set for op replacement so concurrent
-      replicas cannot both accept conflicting replacements for the same nonce.
-- [ ] Support connection options:
-  - [ ] `ERC4337_BUNDLER_STORE_URL` (redis:// / rediss:// / valkey://).
-  - [ ] Sentinel and cluster modes.
-  - [ ] TLS (CA cert, client cert/key), auth (username/password), db index.
-  - [ ] Pool size, timeouts, dial/read/write retries.
-- [ ] Implement `Ping` with a short timeout for readiness.
-- [ ] Encrypt-in-transit (rediss) documented and default in production examples.
+- [x] Add a client dependency (`github.com/redis/go-redis/v9`; Valkey is wire-compatible).
+- [x] Key layout (namespaced by deployment + chain ID):
+  - [x] Mempool op: `...:mempool:{ep}:op:{sender}:{nonce}` → serialized `UserOperation` JSON.
+  - [x] Arrival ordering: `ZSET ...:mempool:{ep}:order` scored by sequence.
+  - [x] Per-sender index: unified entity index (`ZSET ...:mempool:{ep}:entity:{addr}`).
+  - [x] Per-factory / per-paymaster indexes: same `entity` ZSET.
+  - [x] Reputation counters: `HASH ...:reputation:{addr}` (opsSeen, opsIncluded).
+  - [x] Code hashes: `...:checks:codeHashes:{userOpHash}`.
+- [x] Make writes atomic per op using Lua so an op and all its indexes are
+      updated together (replacement of a same-`(sender,nonce)` op does not leak
+      stale index entries; a per-op bookkeeping SET tracks indexed entities).
+- [x] Apply a TTL to pending ops (`ERC4337_BUNDLER_STORE_OP_TTL_SECONDS`, defaulting
+      to `MAX_OP_TTL_SECONDS`) so a crashed replica cannot leave op state forever;
+      stale members are pruned lazily from the indexes.
+- [x] Replacement cleanup is atomic in the same Lua script, so concurrent
+      replacements cannot leak index entries.
+- [x] Support connection options:
+  - [x] `ERC4337_BUNDLER_STORE_URL` (redis:// / rediss:// / valkey://).
+  - [x] Sentinel and cluster modes.
+  - [x] TLS (CA cert, client cert/key), auth (username/password), db index.
+  - [x] Pool size, timeouts, dial/read/write retries.
+- [x] Implement `Ping` with a short timeout for readiness.
+- [x] Encrypt-in-transit (rediss) documented in `docs/CONFIGURATION.md` examples.
 
 ### 1.4 Coordination for horizontal scaling
 
-- [ ] Add a distributed lock manager (Redis lock or Kubernetes Lease):
+- [x] Add a distributed lock manager primitive (`store.Locker`; Redis TTL'd
+      token-checked lock, no-op for single-node backends).
   - [ ] Per-`(ep, sender)` bundle lock so only one replica bundles a sender at a time.
   - [ ] Optional single-active-bundler leader election (simplest correct model);
         followers serve read-only RPCs (`eth_getUserOperationReceipt`, etc.).
@@ -104,19 +114,22 @@ criteria.
 
 ### 1.5 Multi-chain namespacing
 
-- [ ] Prefix every key with chain ID so one Redis/Valkey cluster can serve many
-      chains (`ERC4337_BUNDLER_CHAIN_ID` is implicit today; make it explicit).
-- [ ] Ensure a single process can serve multiple chains, or document the
-      one-process-per-chain topology.
+- [x] Prefix every key with chain ID so one Redis/Valkey cluster can serve many
+      chains (the chain is detected at startup and included in the key prefix).
+- [x] Ensure a single process can serve multiple chains, or document the
+      one-process-per-chain topology (`docs/STORAGE.md` documents the model).
 
 ### 1.6 Acceptance criteria
 
 - [ ] Running tests: kill replica A mid-bundle; replica B reclaims ops after the
-      lease expiry and no op is bundled twice.
-- [ ] Rolling deploy loses zero pending `UserOperation`s.
-- [ ] `redis-cli` shows bounded key growth (TTL enforced) under sustained load.
-- [ ] Readiness reports `store: unavailable` / `503` when Redis is unreachable.
-- [ ] Badger and Redis backends pass the same store test suite.
+      lease expiry and no op is bundled twice. _(Blocked on 1.4 op claims.)_
+- [x] Rolling deploy loses zero pending `UserOperation`s (BadgerDB persistence
+      test; Redis state is shared and TTL-bounded).
+- [x] `redis-cli` shows bounded key growth (TTL enforced): `TestRedisOpTTL`.
+- [x] Readiness reports `store: unavailable` / `503` when Redis is unreachable:
+      `internal/start/health_test.go`.
+- [x] Badger and Redis backends pass the same store test suite
+      (`pkg/store/storetest`), also run against the memory backend.
 
 ---
 

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -38,6 +38,14 @@ services:
       # ERC4337_BUNDLER_IS_OP_STACK_NETWORK: "true"
       # ERC4337_BUNDLER_IS_ARB_STACK_NETWORK: "true"
 
+      # --- Optional: shared storage (multiple replicas) ---
+      # Uncomment these and the `redis` service below to back the mempool with
+      # Redis/Valkey instead of the node-local BadgerDB. See docs/STORAGE.md.
+      # ERC4337_BUNDLER_STORE_BACKEND: "redis"
+      # ERC4337_BUNDLER_STORE_URL: "redis://redis:6379/0"
+      # ERC4337_BUNDLER_STORE_NAMESPACE: "qubenode"
+      # ERC4337_BUNDLER_STORE_OP_TTL_SECONDS: "180"
+
     ports:
       # host:container. In Dokploy you can instead route to container port 4337
       # via the built-in Traefik proxy + a domain and drop this mapping.
@@ -63,5 +71,16 @@ services:
         max-size: "10m"
         max-file: "3"
 
+  # Optional: shared storage for multi-replica deployments. Enable together
+  # with the ERC4337_BUNDLER_STORE_* variables on the bundler service above.
+  # In production prefer a managed Redis/Valkey with TLS and auth.
+  # redis:
+  #   image: valkey/valkey:7-alpine
+  #   restart: unless-stopped
+  #   command: ["valkey-server", "--save", "60", "1", "--appendonly", "yes"]
+  #   volumes:
+  #     - redis-data:/data
+
 volumes:
   bundler-data:
+  # redis-data:
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
@@ -72,6 +72,28 @@ to `v0.9`.
 |----------|---------|-------------|
 | `ERC4337_BUNDLER_OP_LOOKUP_LIMIT` | `2000` | Maximum number of blocks to look back for UserOperation receipts |
 
+### Storage Configuration
+
+The mempool, reputation counters, and code-hash cache run on a pluggable
+backend. See [STORAGE.md](STORAGE.md) for the architecture and scaling model.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `ERC4337_BUNDLER_STORE_BACKEND` | inferred | `badger`, `redis`, `valkey`, or `memory`. Inferred from `STORE_URL` (Redis/Valkey) when empty, else BadgerDB |
+| `ERC4337_BUNDLER_STORE_URL` | None | Connection URL: `redis://`, `rediss://` (TLS), or `valkey://`. Use `rediss://` in production |
+| `ERC4337_BUNDLER_STORE_ADDRS` | None | Comma-separated `host:port` list for Sentinel/Cluster mode |
+| `ERC4337_BUNDLER_STORE_MASTER_NAME` | None | Sentinel master name (enables Sentinel mode) |
+| `ERC4337_BUNDLER_STORE_USERNAME` | None | Redis/Valkey ACL username |
+| `ERC4337_BUNDLER_STORE_PASSWORD` | None | Redis/Valkey password |
+| `ERC4337_BUNDLER_STORE_DB` | `0` | Redis logical database index |
+| `ERC4337_BUNDLER_STORE_POOL_SIZE` | driver default | Connection pool size |
+| `ERC4337_BUNDLER_STORE_NAMESPACE` | None | Deployment prefix added to every key |
+| `ERC4337_BUNDLER_STORE_OP_TTL_SECONDS` | `MAX_OP_TTL_SECONDS` | TTL for pending ops in a shared backend; `0` disables |
+
+The chain ID is always part of the key namespace for Redis/Valkey, so a single
+cluster can safely serve multiple chains. BadgerDB is node-local and keeps the
+legacy un-namespaced layout so existing data keeps loading.
+
 ### Searcher Mode (Required for `--mode searcher`)
 
 | Variable | Default | Description |
@@ -179,6 +201,29 @@ ERC4337_BUNDLER_PRIVATE_KEY=0x...
 ERC4337_BUNDLER_IS_ARB_STACK_NETWORK=true
 ```
 
+### Shared Storage (Redis/Valkey, TLS)
+
+```bash
+ERC4337_BUNDLER_ETH_CLIENT_URL=https://mainnet.infura.io/v3/YOUR_PROJECT_ID
+ERC4337_BUNDLER_PRIVATE_KEY=0x...
+
+# Use a shared backend so several replicas see the same mempool.
+ERC4337_BUNDLER_STORE_BACKEND=redis
+ERC4337_BUNDLER_STORE_URL=rediss://:PASSWORD@redis.example.com:6379/0
+ERC4337_BUNDLER_STORE_NAMESPACE=qubenode-mainnet
+# Defaults to MAX_OP_TTL_SECONDS; set explicitly to bound key growth.
+ERC4337_BUNDLER_STORE_OP_TTL_SECONDS=180
+```
+
+For Sentinel, list the sentinels and the master name:
+
+```bash
+ERC4337_BUNDLER_STORE_BACKEND=redis
+ERC4337_BUNDLER_STORE_ADDRS=sentinel-0:26379,sentinel-1:26379,sentinel-2:26379
+ERC4337_BUNDLER_STORE_MASTER_NAME=mymaster
+ERC4337_BUNDLER_STORE_PASSWORD=...
+```
+
 ### With Observability
 
 ```bash

diff --git a/docs/STORAGE.md b/docs/STORAGE.md
@@ -0,0 +1,114 @@
+# Storage
+
+The bundler keeps three pieces of mutable state: the pending **mempool**, the
+**entity reputation** counters, and the **code-hash cache**. These are accessed
+exclusively through the interfaces in `pkg/store`, so the backing store can be
+swapped without touching the client, bundler, or modules.
+
+## Backends
+
+| Backend | Config value | Shared across replicas | Persistence | Intended use |
+|---------|--------------|------------------------|-------------|--------------|
+| BadgerDB | `badger` (default) | No | On disk | Local dev, single-instance deployments |
+| Redis | `redis` | Yes | Configurable | Horizontally scaled production |
+| Valkey | `valkey` | Yes | Configurable | Wire-compatible with Redis |
+| Memory | `memory` | No | None | Tests, throw-away local runs |
+
+A backend is selected with `ERC4337_BUNDLER_STORE_BACKEND`. When it is left
+empty the bundler infers it: a non-empty `ERC4337_BUNDLER_STORE_URL` implies
+Redis/Valkey, otherwise BadgerDB is used. This keeps existing deployments
+working unchanged.
+
+## Interfaces
+
+```go
+type Store interface {
+    Mempool()    MempoolStore
+    Reputation() ReputationStore
+    CodeHash()   CodeHashStore
+    Locker()     Locker
+    Ping(ctx context.Context) error
+    Close() error
+}
+```
+
+Each concern has its own narrow interface (`pkg/store/mempool.go`,
+`reputation.go`, `codehash.go`). Read operations return `store.ErrNotFound` for
+a missing key and `store.ErrUnavailable` for a backend failure, so callers and
+the readiness probe can tell "absent" from "down".
+
+`Ping` is used by the `/ready` probe. A failure fails readiness (HTTP 503) but
+not liveness, matching the Kubernetes probe contract.
+
+## Key layout
+
+All keys are namespaced by an optional deployment prefix and by chain ID
+(`<namespace>:<chainID>:...`). Leaving both unset reproduces the legacy
+BadgerDB layout so existing data keeps loading. The Redis backend always
+includes the chain ID so one cluster can serve many chains.
+
+| Concern | BadgerDB / memory key | Redis key |
+|---------|-----------------------|-----------|
+| Pending op | `mempool:<ep>:<sender>:<nonce>` | `...:mempool:{<ep>}:op:<sender>:<nonce>` |
+| Arrival order | in-process index | `...:mempool:{<ep>}:order` (ZSET, score = sequence) |
+| Per-entity index | in-process index | `...:mempool:{<ep>}:entity:<addr>` (ZSET) |
+| Entity index bookkeeping | in-process index | `...:mempool:{<ep>}:idx:<sender>:<nonce>` (SET) |
+| Sequence | in-process counter | `...:mempool:{<ep>}:seq` (INCR) |
+| Reputation | `entity:opsCount:<addr>` | `...:reputation:<addr>` (HASH) |
+| Code hashes | `checks:codeHashes:<userOpHash>` | `...:checks:codeHashes:<userOpHash>` |
+| Locks | not shared | `...:lock:<name>` |
+
+The `{<entryPoint>}` hash tag makes every key touched by a single mempool
+operation hash to the same Redis Cluster slot, so the atomic scripts below are
+cluster-safe.
+
+## Atomicity and TTL
+
+Adding or removing a Redis op is a single Lua script that writes the op value,
+the arrival-order ZSET, every entity ZSET, and a small bookkeeping SET together.
+Replacing an op with the same `(sender, nonce)` first removes the previous
+factory/paymaster index entries, so a swapped paymaster cannot leave a stale
+index behind. Reputation updates for a single logical event (for example one op
+touching a sender, a factory, and a paymaster) are applied in one script so the
+counters stay consistent.
+
+Pending ops in a shared backend are given a TTL
+(`ERC4337_BUNDLER_STORE_OP_TTL_SECONDS`), which defaults to
+`ERC4337_BUNDLER_MAX_OP_TTL_SECONDS`. An op whose value key expires is pruned
+from the ordered indexes lazily on the next read, so a crashed replica cannot
+leave op state behind forever and key growth stays bounded.
+
+## Scaling model
+
+The store abstraction makes the bundler's state external, which is a
+prerequisite for running more than one replica. Two deployment topologies are
+supported:
+
+- **Single active bundler + read replicas.** The simplest correct model. Only
+  one process runs the batch loop; additional replicas serve read-only RPCs
+  (`eth_getUserOperationReceipt`, `eth_getUserOperationByHash`). No coordination
+  is required.
+- **Active/active.** Multiple processes share the mempool. Correctness requires
+  that two replicas never build overlapping bundles for the same
+  `(entryPoint, sender, nonce)`. The backend exposes a `Locker`
+  (`Store.Locker()`): Redis implements a TTL'd, token-checked distributed lock,
+  and single-node backends return a no-op locker so callers can depend on the
+  interface unconditionally.
+
+> **Status:** the lock manager and op TTL are in place. Wiring per-sender bundle
+> locks, op leases, and leader election into the batch loop is the next step
+> (roadmap section 1.4); until then, use the single-active-bundler topology for
+> a shared store.
+
+## Readiness
+
+`/ready` reports the upstream node (`eth_client`) and the storage backend
+(`store`). When the backend cannot be reached the response is `503` with
+`"store": "unavailable"`, so load balancers drain the instance without the
+orchestrator restarting it.
+
+## Backward compatibility
+
+`ERC4337_BUNDLER_DATA_DIRECTORY` continues to work: with no store URL the
+bundler opens a BadgerDB in that directory using the original key layout, so an
+existing deployment upgrades in place without losing pending ops.