Repository navigation
Conversation
A pool cannot yet tell whether GPU partitioning is in use on a host,
being entered or left, or why a host does not take part. Nothing
records it, so nothing can check it before enabling partitioning, and
an administrator cannot see it.
Each host now reports its own state in two read-only fields, shown by
xe host-param-list: a status and the reason for it. Later changes that
add the pool-wide switch and the steps that enter and leave
partitioning read and extend them.
Host.gpu_partitioning_status has all five values from this release on,
as a host fails on a stored enum value it does not know; new causes go
in Host.gpu_partitioning_reason, an open set of tokens. At xapi start,
after the database sync and on its own thread, the host runs its checks
under one lock, most specific first, and writes both fields together.
NVIDIA cards are found by PCI vendor: pass-through unbinds the driver.
Nothing else changes: the internal Host.create takes both fields, with
defaults; no VM, GPU or placement path reads them; and the GPU
compatibility check keeps its vendor test. Until the pool can enable
partitioning, a host shows inactive once its xapi has started on this
version; an older host, or a joining one until its xapi restarts,
shows not_reported. Schema bumped; tests cover the checks and the write.
After xapi starts, until the pool can enable partitioning, every host
logs (debug) its state and xe shows it; on a host with an NVIDIA card:
Xapi_host.sync_gpu_partitioning: inactive (pool_disabled)
$ xe host-param-list uuid=<host> | grep gpu-partitioning
gpu-partitioning-status ( RO): inactive
gpu-partitioning-reason ( RO): pool_disabled
and on a host without one, the reason is no_partitionable_gpu.
Signed-off-by: Marcus Granado <marcus.granado@citrix.com>
A helper program is to report how an NVIDIA card is partitioned (MIG mode, GPU instances, the profile catalogue, the vGPUs placed on them) as one JSON reply, and through --version which commands and format versions it serves. xapi runs it and reads both replies. Both sides need one description of these replies, and xen-api has none. Add the replies' types, with their JSON readers and writers, to the gpumon interface, which gpumon and xapi already link, in sub-modules named after what they describe. A later minor version may add members and values, and these types must still read its replies. gpumon's existing calls and their JSON are unchanged; nothing reads the new types until xapi's readers of the replies are added. Each record has the reply's member names. An enumeration value that this reader does not know is kept as Unknown, and only the exact lower-case spelling reads as Known, since rpclib's variant reader ignores case. A member that is always present but may be null has its own typ. A member the reader can do without is kept as an Error when it is absent or cannot be read, and an integer must be a JSON integer in range. A profile's signature is the resource counts, memory size and capabilities that the driver reports for it. It tells profiles apart beyond their names and numbers, and can show when a driver renumbers them. A member kept as an Error is lenient, so that a fault in it costs only that member, not the whole reply. The gpumon interface test builds sample replies as OCaml values, writes each one and reads it back. Further cases edit a written reply, to drop or break a member or add what a newer helper may send, and check that the reader refuses the reply or loses only that member. Single values check the enumerations' spelling and the one renamed member. Signed-off-by: Marcus Granado <marcus.granado@citrix.com>
PGPU.partition_mode can say whether a card is divided into partitions,
but nothing sets it from the card: xapi cannot yet read the reply of
the GPU partition helper, a program that reads the partitions of one
card, although the reply's types exist.
A host taking part in GPU partitioning will read, at xapi start, each
of its NVIDIA GPUs bound to the nvidia driver, and write what it finds,
so an administrator sees the state of each card without acting.
The decoder reads the reply on the shared types: the mode and the case
(partitions, none carved, unreadable with its reason, not applicable)
strictly, other sections leniently, pages folded into one reply. A
reason is the helper's kind, or one of xapi's own when there is no
usable reply. Each card reads unknown, then is written on its answer;
a read that fails in any other way writes its card unknown and logs why.
The helper's path is the xapi.conf key gpu-partition-helper, by default
/opt/xensource/libexec/gpu-partition-helper; a missing one is never
forked, and a run has a 5 s deadline. Its lock arguments are passed
apart, for a later lock, and Partition_mode now exports axis. Tests
write typed replies and edit the written value or text, one named case
per purpose. Until the pool can enable partitioning, no helper runs.
After xapi starts, every host logs (debug) the new key, which
/etc/xapi.conf lists commented out:
gpu-partition-helper = /opt/xensource/libexec/gpu-partition-helper (Reads the partitions of the host's GPUs)
Signed-off-by: Marcus Granado <marcus.granado@citrix.com>
PGPU.partition_mode reads unknown on every new GPU and cannot say that
partitioning is not in use on its host, that xapi does not read this
GPU by design, or that a GPU once divided has not been read since.
At xapi start, a failure that holds for the whole host, such as a
missing helper or driver, would be found again on each GPU in turn.
Each GPU now says whether partitioning applies to it, so an
administrator sees why a GPU is not read, and placement can later keep
new work off a GPU that may still be divided.
Three values join the field in its first release, as a host fails on a
stored enum value it does not know: not_checked; not_in_use, now the
default; possibly_enabled, which a failed read writes over a division.
The first start after a boot sets an inactive host's GPUs to not_in_use
unless possibly divided. A host taking part marks GPUs it does not read
not_checked; GPUs not yet read take a host-wide failure's reason only.
The boot reset applies at once: no host takes part yet. The read, its
new values and not_checked wait until the pool can enable partitioning:
no change in behaviour until then. The host sync is told if this is
the first start after a boot, so no later caller repeats the reset.
Schema bumped; the compiler checks every reason is classified. The
tests add one named case per purpose to the existing lists.
After the next reboot, each GPU of a host reads not_in_use in the API
field PGPU.partition_mode, and a host with an NVIDIA GPU logs (debug):
Xapi_host.sync_gpu_partitioning: inactive (pool_disabled), after boot: true
and "after boot: false" on a toolstack restart, which changes no GPU.
Signed-off-by: Marcus Granado <marcus.granado@citrix.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for now because this PR depedns/based on #7377 and #7378 (the helper's reply types in
Gpumon_interface). Until they merge, this PR also shows their two commits; only the last commit (gpu_partition_helper) is new here.PGPU.partition_modecan say whether a GPU is divided into partitions, but nothing sets it from the card yet. This PR adds the first reader. At xapi start, a host taking part in GPU partitioning runs the GPU partition helper (a program that reads the partitions of one card) for each of its NVIDIA GPUs bound to thenvidiadriver, and writes what it finds.Gpu_partition_probedecodes the helper's reply with the reply types inGpumon_interface. It reads the mode and the case strictly (partitions, none carved, unreadable with a reason, not applicable) and every other section leniently. Pages are folded into one reply. A reason is either the helper's own kind (Helper) or one of xapi's (Xapi) when there is no usable reply: helper absent or not executable, timeout, crash, undecodable output, card busy. When a reply cannot be decoded, the reader's text is logged at debug.Gpu_partition_helperruns the helper withHelpers.call_scriptand a 5 s deadline. The path is the newxapi.confkeygpu-partition-helper, by default/opt/xensource/libexec/gpu-partition-helper; a missing helper is never forked. Each card is first writtenunknown, then given its answer. A read that fails in any other way writes only that cardunknown, logs why, and the other cards are still read. The helper's argument list is data, so that a later change can wrap the call in a lock.Partition_modenow exportsaxis.Caller:
Xapi_host.sync_gpu_partitioning, in the arm where the host takes part in partitioning. Unchanged behaviour: that arm cannot run until the pool can enable partitioning, so for now no helper runs and nothing writespartition_mode. The only new output is the start-up dump line of the new key:gpu-partition-helper = /opt/xensource/libexec/gpu-partition-helper (Reads the partitions of the host's GPUs).Tests:
dune buildofocaml/xapi,ocaml/libs/gpuandocaml/tests/suite_alcotest.exe, anddune runtest ocaml/testspass, 60 of them inTest_gpu_partition_helper. The release-profile build anddune build @fmtare also clean. The tests have no data files. Each test builds typed replies, writes them with the deriver, and edits the written value or text to make the broken cases. Every case has its own name, which says what it checks.