Skip to content

About

Upgrades DRBD on hypervisors orchestrated with Apache CloudStack, Proxmox VE, OpenStack, OpenNebula, or XCP-ng/XOSTOR.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

drbd-rolling-upgrade

Upgrade DRBD across your cluster one node at a time, without ever taking down the only copy of live data.

You point it at a cluster whose guests are managed by CloudStack, Proxmox VE, OpenStack, OpenNebula or XCP-ng (XOSTOR). For each node it:

  1. checks the LINSTOR placements and adds a temporary replica wherever the node holds the only copy, or a vote that quorum needs;
  2. live-migrates the guests off the node and puts it into the orchestrator's maintenance mode;
  3. moves drbd-reactor promoters away;
  4. installs the new DRBD packages while the old module keeps serving, then swaps the kernel module and checks that every resource came back;
  5. takes the node out of maintenance, and moves on to the next.

When something looks wrong, the tool stops and tells you what it found and what to do. It does not try to repair a cluster.

Install

The tool is pure Python (3.6 or newer), standard library only. It uses python-linstor if it is installed, and otherwise calls the linstor client.

pip install .                 # installs the drbd-rolling-upgrade command

Or build a .deb or .rpm: see packaging/README.md.

Install it on a control host. The tool reaches the nodes over the root ssh mesh your cluster already has, and it must not run on a node it is about to upgrade.

Before you start

  • Run it inside tmux or screen. The steps on the nodes survive a dropped connection, but the tool itself does not.
  • Make sure the linstor client works on the control host. On a controller node its configuration is usually root's: add --linstor-sudo.
  • If you pin the DRBD packages (dnf/yum versionlock, apt-mark hold, zypper locks), clear the pin on a node right before its turn and set it again afterwards. The tool never clears a pin for you; it stops and names it.
  • Your orchestrator's section in docs/OPERATIONS.md says which credentials and options the tool needs there.

Use

Start with the read-only commands. They change nothing and are safe to run at any time:

drbd-rolling-upgrade --linstor-sudo analyze
drbd-rolling-upgrade --linstor-sudo preflight --target-version 9.3.4

run shows its plan and changes nothing unless you add --execute:

drbd-rolling-upgrade run --target-version 9.3.4 --state-file /root/dru.json
drbd-rolling-upgrade run --target-version 9.3.4 --state-file /root/dru.json --execute

If a run stops, fix what it reported and continue where it left off:

drbd-rolling-upgrade resume --execute --state-file /root/dru.json

When all nodes are done, remove the temporary replicas it added:

drbd-rolling-upgrade post-upgrade --execute --state-file /root/dru.json

Confirmations

Every step that changes something asks you first. To let a run go through without asking, pre-approve classes of steps with --yes <class>, or --yes all. drbd-rolling-upgrade gates lists the classes.

Two steps always ask, whatever you pass: rebooting a node (needed only when the old DRBD module cannot be unloaded) and deleting a replica. Run the tool attached to a terminal whenever it may reach either of them. Without a terminal to answer, the answer is "no": the run stops, and you can resume it later.

Rehearsing

You can try the analysis against a saved LINSTOR state instead of a live cluster:

linstor -m --output-version v1 resource list > resources.json
drbd-rolling-upgrade --linstor-json resources.json analyze

Tested on

Each of these clusters had a full rolling upgrade through the tool, with guests running and live-migrated, plus a rollback of one node and forward again:

Orchestrator Nodes OS and DRBD packages DRBD
CloudStack 4.23 5 AlmaLinux 10, dnf, kmod-drbd 9.3.3 → 9.3.4
Proxmox VE 8.4 3 Debian 12, apt, drbd-dkms 9.3.3 → 9.3.4
OpenStack 2025.1 (kolla) 3 Ubuntu 24.04, apt, prebuilt drbd-module-<kernel> 9.3.3 → 9.3.4
OpenStack (Platform9 Private Cloud Director) 3 Ubuntu 22.04, apt, drbd-dkms 9.3.3 → 9.3.4
OpenNebula 7.2 3 AlmaLinux 9, dnf, kmod-drbd 9.3.3 → 9.3.4
XCP-ng 8.3 with XOSTOR 3 dom0, yum, XOSTOR's and LINBIT's kmod-drbd 9.2.16 → 9.2.18 → 9.3.3 → 9.3.4

Also exercised on real clusters: package pins (versionlock, apt-mark hold), drbd-reactor moving the LINSTOR controller during a run, the orchestrator's management node being one of the upgraded nodes, and a reboot through the reboot confirmation (Platform9, XCP-ng).

Not tested on a real cluster yet: SLES (zypper), Proxmox VE 9, Proxmox's HA manager with HA resources, and on XCP-ng a reboot of the pool master and pool HA.

More

docs/OPERATIONS.md is the full guide, with a section for each orchestrator, every confirmation step, what each halt means, rollbacks, and resume.

Development

python3 -m pytest

tools/ holds scripts that build test fleets of guests and LINSTOR resources on a lab cluster, one per orchestrator; each has a FLEET*.md next to it. docs/STATUS.md records what has been verified where, in detail, and what is still open.

About

Upgrades DRBD on hypervisors orchestrated with Apache CloudStack, Proxmox VE, OpenStack, OpenNebula, or XCP-ng/XOSTOR.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages