Upgrade DRBD across your cluster one node at a time, without ever taking down the only copy of live data.
You point it at a cluster whose guests are managed by CloudStack, Proxmox VE, OpenStack, OpenNebula or XCP-ng (XOSTOR). For each node it:
- checks the LINSTOR placements and adds a temporary replica wherever the node holds the only copy, or a vote that quorum needs;
- live-migrates the guests off the node and puts it into the orchestrator's maintenance mode;
- moves drbd-reactor promoters away;
- installs the new DRBD packages while the old module keeps serving, then swaps the kernel module and checks that every resource came back;
- takes the node out of maintenance, and moves on to the next.
When something looks wrong, the tool stops and tells you what it found and what to do. It does not try to repair a cluster.
The tool is pure Python (3.6 or newer), standard library only. It uses
python-linstor if it is installed, and otherwise calls the linstor client.
pip install . # installs the drbd-rolling-upgrade command
Or build a .deb or .rpm: see packaging/README.md.
Install it on a control host. The tool reaches the nodes over the root ssh mesh your cluster already has, and it must not run on a node it is about to upgrade.
- Run it inside
tmuxorscreen. The steps on the nodes survive a dropped connection, but the tool itself does not. - Make sure the
linstorclient works on the control host. On a controller node its configuration is usually root's: add--linstor-sudo. - If you pin the DRBD packages (
dnf/yum versionlock,apt-mark hold,zypper locks), clear the pin on a node right before its turn and set it again afterwards. The tool never clears a pin for you; it stops and names it. - Your orchestrator's section in
docs/OPERATIONS.mdsays which credentials and options the tool needs there.
Start with the read-only commands. They change nothing and are safe to run at any time:
drbd-rolling-upgrade --linstor-sudo analyze
drbd-rolling-upgrade --linstor-sudo preflight --target-version 9.3.4
run shows its plan and changes nothing unless you add --execute:
drbd-rolling-upgrade run --target-version 9.3.4 --state-file /root/dru.json
drbd-rolling-upgrade run --target-version 9.3.4 --state-file /root/dru.json --execute
If a run stops, fix what it reported and continue where it left off:
drbd-rolling-upgrade resume --execute --state-file /root/dru.json
When all nodes are done, remove the temporary replicas it added:
drbd-rolling-upgrade post-upgrade --execute --state-file /root/dru.json
Every step that changes something asks you first. To let a run go through
without asking, pre-approve classes of steps with --yes <class>, or
--yes all. drbd-rolling-upgrade gates lists the classes.
Two steps always ask, whatever you pass: rebooting a node (needed only
when the old DRBD module cannot be unloaded) and deleting a replica. Run the
tool attached to a terminal whenever it may reach either of them. Without a
terminal to answer, the answer is "no": the run stops, and you can resume
it later.
You can try the analysis against a saved LINSTOR state instead of a live cluster:
linstor -m --output-version v1 resource list > resources.json
drbd-rolling-upgrade --linstor-json resources.json analyze
Each of these clusters had a full rolling upgrade through the tool, with guests running and live-migrated, plus a rollback of one node and forward again:
| Orchestrator | Nodes | OS and DRBD packages | DRBD |
|---|---|---|---|
| CloudStack 4.23 | 5 | AlmaLinux 10, dnf, kmod-drbd |
9.3.3 → 9.3.4 |
| Proxmox VE 8.4 | 3 | Debian 12, apt, drbd-dkms |
9.3.3 → 9.3.4 |
| OpenStack 2025.1 (kolla) | 3 | Ubuntu 24.04, apt, prebuilt drbd-module-<kernel> |
9.3.3 → 9.3.4 |
| OpenStack (Platform9 Private Cloud Director) | 3 | Ubuntu 22.04, apt, drbd-dkms |
9.3.3 → 9.3.4 |
| OpenNebula 7.2 | 3 | AlmaLinux 9, dnf, kmod-drbd |
9.3.3 → 9.3.4 |
| XCP-ng 8.3 with XOSTOR | 3 | dom0, yum, XOSTOR's and LINBIT's kmod-drbd |
9.2.16 → 9.2.18 → 9.3.3 → 9.3.4 |
Also exercised on real clusters: package pins (versionlock, apt-mark hold),
drbd-reactor moving the LINSTOR controller during a run, the orchestrator's
management node being one of the upgraded nodes, and a reboot through the
reboot confirmation (Platform9, XCP-ng).
Not tested on a real cluster yet: SLES (zypper), Proxmox VE 9, Proxmox's HA manager with HA resources, and on XCP-ng a reboot of the pool master and pool HA.
docs/OPERATIONS.md is the full guide, with a section for each orchestrator,
every confirmation step, what each halt means, rollbacks, and resume.
python3 -m pytest
tools/ holds scripts that build test fleets of guests and LINSTOR resources
on a lab cluster, one per orchestrator; each has a FLEET*.md next to it.
docs/STATUS.md records what has been verified where, in detail, and what is
still open.