Skip to content

Latest commit

 

History

History
213 lines (182 loc) · 15.3 KB

profiling.md

File metadata and controls

213 lines (182 loc) · 15.3 KB

Measure CPU cost on your OP-XY

New or changed draft and published catalog entries must include a hardware cpu measurement for their pinned machine commit, with both average CPU and worst observed block CPU (peak/spike). Profiling is complete only after the generated cpu member is added to the catalog entry. This applies to synths, drum kits, sample engines and effects. Unchanged older entries can remain without one until their next update. A maintainer can run a batch for contributors who do not have hardware; contributors can also follow this procedure themselves. The host estimate is useful during development but cannot fill this field; it also swings several-fold while other builds or tests run on the same computer, so time one machine at a time. CI validates the record and revision; it cannot independently attest that a contributor ran the hardware test.

Build and install

Use the SDK's staging branch and commit your machine changes before building. Follow building firmware to install the pinned tools and obtain your own stock OS. Keep firmware and packages outside every checkout. In the commands below, <private> is that external folder and <machine> is your machine folder.

python -m pip install python-rtmidi
python sdk.py build --machine <machine> --profiling on --stock-os <private>/stock-1.2.1/os.ldr --out <private>/cpu-build-001

Profiling builds sweep synth controls by default: after each synth's standard run, the firmware repeats the stock workload once per setting, each control in turn through nine positions (or each option of a list) with the others at their defaults, then once with every control at the position that ran heaviest. A synth's catalog record is that sweep. The standard run is stock's own: controls 0-3 at midpoint, any others as the track holds them; swept runs set all eight. A sweep build's voices also check, once per block, whether a sweep run is theirs, a few instructions beside a voice's thousands. --sweep off gives a quick midpoint-only build for development; it cannot publish a synth record.

Repeat --machine or use --collection to build a batch. Keep the package: its manifest and compile report identify the firmware and expected test order. Install its candidate/ files using the installation steps, write the OS to the boot partition, and power-cycle. The bootloader is unchanged. Website builds disable the required per-voice counters; use the SDK build above.

Boot to the normal instrument screen, with USB connected. MTP mode does not provide the MIDI endpoint. Save your project and stop playback: this is an active diagnostic that plays its own notes and changes temporary instrument settings. Close applications holding the XY's MIDI ports and leave USB audio off. The collector needs exactly one OP-XY MIDI input/output pair. It uses USB MIDI, not a serial/CDC port. No additional dev-menu toggle is needed after booting the installed development image normally.

Collect a run

One capture is enough: each run averages 1,500 blocks of a fixed workload, and repeated captures of the same build agree to within a few hundredths of a percent. A machine with randomness (noise, random modulation) may vary more: collect several, one after another, each to a new directory, and pass every capture to the report. For averages it publishes their median (two give their average); for peaks it keeps the largest observed block across every supplied capture and every measured setting/workload. Never discard a capture because its spike is higher. One capture meets the submission requirement but cannot prove an absolute worst-case bound. Each stock-workload run takes about 2.2 seconds, so a sweep run takes longer: about 80 seconds for a synth with four controls. The collector waits for every expected report (by default up to 3 seconds per run plus 15) and stops early once they have arrived.

python sdk.py profile benchmark --package <private>/cpu-build-001 --installed --greet --out <private>/cpu-run-001

Never run two collectors at once: they share the OP-XY's one MIDI port pair. Restart the OP-XY normally, without saving, before every run after the first: later runs in the same boot are counted twice, and the report refuses their capture (Report count differs from this build). Waiting between runs doesn't help. A swept setting that times nothing (a level at 0) is left out only when two or more captures agree it is silent, since a failed run reads silent too: profile report names it and asks for a second capture.

--installed confirms that this exact package is on the device and your work is saved. Metadata identifies normal OS 1.2.1 but does not attest the installed image hash. The collector checks CRCs, endpoint, packet size, and the complete ordered report count before assigning results to machines. A missing or failed registration cannot silently shift the reported names. Do not operate the instrument while a run is active. Each command waits up to 60 seconds for data.

Each run saves capture.json, raw traffic.jsonl, and a readout. Keep these private: device metadata can contain a serial number. The command installs no firmware and never reboots the device. Restart the instrument after testing before resuming normal work; do not save the diagnostic's temporary settings.

For connection diagnosis, python sdk.py profile probe --greet --out <fresh-folder> requests metadata without running the benchmark. profile selftest exercises the protocol codec without opening a device. An inactive/failed workload, missing reports, or disabled profiling is a failed measurement, never a zero-cost machine.

Add the catalog field

The following is offline; --machine here takes the machine ID, not its folder. It verifies that the clean, committed machine source still matches the measured package. Existing complete captures already contain the peak: rerun this command with the current SDK and the original package/captures to upgrade an average-only report. No new build or device run is needed when those inputs still match. Repeat --capture for every independent capture of that build.

python sdk.py profile report --package <private>/cpu-build-001 --machine mymachine --capture <private>/cpu-run-001/capture.json --out <private>/cpu-report-001

Copy the cpu member from catalog-cpu.json into your machine's entry in catalog/machines.json. To link evidence as well, publish the generated report.md (an issue, PR comment or file in your repository) and pass its URL with --report-url; if you only learn a comment's permalink after posting, rerun the offline report command into a new directory with it. No new device run is needed. The report omits serial numbers and local paths. Share the raw captures privately with a maintainer if needed; never publish firmware or firmware-containing packages.

cpu field Meaning
benchmark Versioned workload from the table below; must match the machine kind
commit Full machine source SHA, equal to the catalog entry's commit (or an earlier release's, when a maintainer approves carrying it over)
firmware_sha256 Exact tested candidate OS image
runs 1 to 100 independent, complete captures of the same build and collector (one is enough)
block_percent The run's mean cost (the median across captures when there are several), as a percentage of the stock 580,352-cycle block budget. For a synth, the largest median across midpoint, swept and combined runs; for a drum kit, the largest single-voice median. This is an average, not a spike
peak_percent Required for new measurements: the largest single measured block across all settings/workloads and all supplied captures, in the same budget units. For drums this includes whole-kit, choke and ceiling runs. Never a median of maxima or the mean of the heaviest setting. The generated report identifies the workload and shows each run's peak
voice_peak_percent Drum kits, optional (written by profile report since the CPU-high label): the largest single block across the single-voice runs only, every model and level and every capture, leaving out the whole-kit, choke and ceiling runs. At least block_percent, at most peak_percent
midpoint_percent Synths: the midpoint run's median, stock's own workload
combined_percent Synths: the median of the run with every control at its heaviest position
worst_setting Synths: the heaviest swept setting: control (0-7), its label, and its Q31 value
report Optional: public HTTPS evidence URL containing the generated readout

Keep both block_percent (average) and peak_percent (worst observed block) at or below the recommended 3.5% each. A higher result is still accepted, but profile report, the collector's readout and catalog check warn for each metric above the recommendation. The report identifies the workloads: make those paths lighter where you can. This is guidance, not a hard admission limit. Inspect both metrics; a low mean can conceal an expensive block. Both timing fields use 100 * encoded_value / 65535. A peak encoded as 65535 is clipped at the whole block budget: the collector labels it at least 100%, and profile report refuses to publish it as an exact peak. Optimize the named path and remeasure; do not replace the clipped value with an estimate or omit that capture. Invalid peaks below their run's mean are also refused. Stock keeps rendering a voice until its amplitude envelope ends, so a long release calls render well after the machine's own sound has died away, and the benchmark's 500 release blocks count that work. Once a voice's tail is below an inaudible floor and nothing more will sound, write silence and skip the rest.

A drum kit's record has a run per model and level, plus three for the whole kit, so a kit of 33 models adds about 100 reports and takes about half a minute. block_percent is its heaviest single voice; kit_percent, choke_percent and ceiling_percent are whole kits against the same budget. peak_percent covers all those workloads, so its warning may refer to a whole-kit run. Use the report's workload labels when interpreting it; a whole-kit peak is not a per-voice cost.

Synth records from before the sweep (opxy-synth-c8-v1) stay valid while their release stays at the commit they measured; a new synth, or one whose release moves, submits a sweep (unless a maintainer approves carrying the record over). Average-only records may likewise stay unchanged, including an evidence-link update or promotion from draft at the same commit. New measurements require peak_percent; a changed machine revision needs new measurements unless a maintainer explicitly approves carrying the exact previous record over.

Updating a machine's commit requires a new measurement, unless a catalog maintainer approves keeping the previous one (catalog updates). CI rejects stale results, host estimates, non-finite/zero/saturated values and missing records on changed submissions. Retiring an entry does not require a new run. Validate your submission:

python sdk.py catalog check --base origin/staging

What is measured

All workloads run on the Blackfin handling the diagnostic USB-MIDI command. They are synthetic measurements on real hardware, not passive measurements of the audio core during normal project playback. Every block is 64 samples.

Machine kind / benchmark Scope and fixed workload
Synth / opxy-synth-c8-sweep-v1 Stock whole-voice timer, including oscillator and filter. Stock diagnostic plays note 60 at velocity 127 for 1,000 blocks and releases for 500, first with controls 0-3 at midpoint (stock's own run, opxy-synth-c8-v1), then once per swept setting: one control at a time through nine positions from 0 to full scale (each option of a list), the others at their defaults, as the SDK's host estimate sweeps them, then once with every control at the position that ran heaviest in that capture (the firmware picks them from stock's timing totals; the readout names them from the same capture's reports). The voice adapter renders with the swept controls while a sweep run is active. Other combinations of controls are not run.
Effect / opxy-effect-callback-v1 One stereo process callback at declared defaults for 1,500 blocks. Deterministic triangle/impulse input, separate initialized state, zeroed private scratch buffer when declared, 120 BPM transport. Includes the wet algorithm, not the stock send/mix path.
Drum kit / opxy-drum-callback-v1 The kit's voices through the SDK's drum adapter (+20 start and +12 render), with a stored key's unpacking, offsets, amplitude and same-key choke, on 1,500 blocks each, a note's start timed with its block. One voice of every model with all its controls at 64, 0 and 127; the kit's own keys at voiceCapacity voices, saved at their defaults, as a project's are, struck every 100 blocks; two voices retriggering one key every 8 blocks with the heaviest of those; and voiceCapacity voices of the heaviest, each on a key of its own. Excludes stock's drum renderer, envelope and mixing, so it isn't a whole-voice figure as a synth's is
Sample / opxy-sample-callback-v1 One render_sample callback at declared defaults for 1,500 blocks. A generated two-second, mono 44.1 kHz PCM triangle/impulse sample, full region, unity level, forward playback, C4 pitch, 120 BPM, no transpose; note restarts every 500 blocks. Declared sample analysis and assignment run before timing. Output uses the machine's mono/stereo contract.

Sample and effect tests allocate their own buffers during setup, outside audio callbacks, and free them afterward. They do not borrow a project's assigned sample, live voice or send scratch buffer. Failed allocation/preparation or an entirely silent fixture yields an invalid measurement. Their stopwatch includes counter-call overhead and may include interrupt time; the stock synth runner masks interrupts while rendering. Do not compare different scopes as though they include the same work.

100% is the stock diagnostic's normalization budget, not total device utilization. Neither the mean nor the worst observed block is a proven worst-case bound or a maximum-polyphony prediction. The synth's combined setting is selected by mean cost, not by peak, and the sweep does not exhaust all control combinations. Graphics, project scheduling, disk/USB I/O and sample preparation are excluded. Check demanding settings, control motion, many simultaneous voices, feedback/tails and normal playback separately. A successful benchmark is not full hardware acceptance.

Agents and maintainers

sdk.py test, check, build and catalog CI never contact a device. Hardware profiling is an explicit task: confirm the user wants the device test, prepare and verify a local build, install only when instructed, then wait for the user to flash/reboot before collecting. Do not infer installed firmware from a package path or turn host timings into hardware claims. The same batch captures can produce one offline report per included machine. Before declaring submission ready, generate the report, explain both mean and peak and their measured scope, and add the complete cpu member to the catalog entry. If hardware or valid captures are unavailable, leave the entry reserved and report profiling pending; do not fabricate evidence or weaken the catalog check.