Repository navigation
Storage fixes - #7369
Open
last-genius wants to merge 3 commits into
Open
Storage fixes#7369last-genius wants to merge 3 commits into
last-genius wants to merge 3 commits into
Conversation
We've seen a case where multiple VDI.deactivate calls running in parallel get into a race between get_other_config and add_other_config, failing with the Duplicate_key exception. If one of the calls succeeded in adding other_config=content_id:, then simply continue. Signed-off-by: Andrii Sultanov <andriy.sultanov@vates.tech>
gthvn1
reviewed
Nov 4, 2026
gthvn1
left a comment
gthvn1
left a comment
Contributor
There was a problem hiding this comment.
Just a remark, otherwise LGTM
gthvn1
approved these changes
Nov 4, 2026
last-genius
force-pushed
the
asv/storage-fixes
branch
from
November 4, 2026 13:13
2752ef4 to
b2df4c5
Compare
XCP-ng uses VBDs in dom0 for some use-cases, and failures can result in
leftover datapaths breaking unrelated VBDs.
To help avoid this, if attach succeeded but activate failed, detach
the dom0 VBD. This requires reordering the functions so that
attach_and_activate can call deactivate_and_detach.
Details of this issue:
This script can be used to check for leftover datapaths:
$ cat check_datapath.py
import json
d = json.load(open('/var/run/nonpersistent/xapi/storage.db'))
claims = {}
for sr, srv in d['host']['srs'].items():
for vdi, vv in (srv.get('vdis') or {}).items():
for dp, state in (vv.get('dps') or {}).items():
claims.setdefault(dp, []).append((sr, vdi, state))
for dp, owners in sorted(claims.items()):
print(dp, ' <-- DUPLICATE' if len(owners) > 1 else '')
for sr, vdi, state in owners:
print(' ', sr, vdi, state)
The following sequence simulates a failure to activate a VBD and shows how two
VBDs using the same datapath results in further errors down the line:
$ touch /tmp/fist_blktap_activate_inject_failure
$ xe vbd-plug uuid=$FIRST_VBD_UUID
Error code: SR_BACKEND_FAILURE_1200
Error parameters: , injected failure,
$ rm -f /tmp/fist_blktap_activate_inject_failure
$ python3 check_datapath.py
vbd/0/xvda
1d4a39c0-5029-a134-77c1-b189d2e60dfc 6016c6ad-1101-4d0a-bc32-bc9604d930ab ['Attached', 'RW']
$ xe vbd-destroy uuid=$FIRST_VBD_UUID
$ xe vbd-plug uuid=$SECOND_VBD_UUID
$ python3 check_datapath.py
vbd/0/xvda <-- DUPLICATE
1d4a39c0-5029-a134-77c1-b189d2e60dfc 7ceaa7c6-4c51-4536-9c84-7d25d11e3698 ['Activated', 'RW']
1d4a39c0-5029-a134-77c1-b189d2e60dfc 6016c6ad-1101-4d0a-bc32-bc9604d930ab ['Attached', 'RW']
$ xe vbd-unplug uuid=$SECOND_VBD_UUID
message: Expected 0 or 1 VDI with datapath, had 2
Only reboots will fix this, otherwise the storage.db needs to be manually
modified to drop the stale entry.
Signed-off-by: Andrii Sultanov <andriy.sultanov@vates.tech>
last-genius
force-pushed
the
asv/storage-fixes
branch
from
November 4, 2026 14:22
b2df4c5 to
4c0ca01
Compare
Previous commit fixed one of the causes of "stale" VDIs taking up datapaths: activation failure. There are, however, probably more ways these can happen. Additionally, already existing stale datapaths need to be cleaned up. This commit changes the clean-up of leaked DPs that happens before VDI.attach. The function now also cleans up a stale datapath used by a different VDI (a stale datapath is defined as that which has been attached but not activated) before re-using this datapath for a new VDI. This is done at attach time to cover all possible reasons for occurrence of such a stale datapath. The clean-up needs to be done before the new datapath is attached, since the rest of the module assumes only 0 or 1 VDIs use a datapath at the same time. This should be safe to do, since xapi already let the VBD be created (so it thinks its device is not in use), and the datapath is not activated. This requires refactoring the function to handle removing datapaths for different VDIs, not just the one we are creating a datapath for. Also specify the exact reason the datapath is being removed to enable easier debugging in case of an issue. Signed-off-by: Andrii Sultanov <andriy.sultanov@vates.tech>
Contributor
Author
|
Added another commit - this one to clean up a stale datapath before re-using it for a different VDI. This should hopefully cover all bases, so that the issue of stale datapaths does not result in two VDIs using the same datapath. |
Contributor
Author
|
I tested the last commit manually following the sequence in the second commit (with the second commit dropped so that it doesn't interfere). There are no 'duplicate' datapaths anymore and no more of the "expected 0 or 1, got 2" error. The log shows that the stale datapath was cleaned up: |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First commit fixes multiple
VDI.deactivaterunning in parallel.Second commit cleans up on
VBD.plugfailure fordom0VBDs.Third commit cleans up stale datapaths before re-using them.