console: add resource usage charts to the redesigned cluster page

[leedqin]

Motivation

Third PR in the cluster details redesign stack, on top of #39507. It replaces the Metrics tab's overview charts with the resource usage card from the design. CPU, memory and replica status share one time axis and one tooltip, so a spike, an out-of-memory kill and the restart after it line up. Fixes CNS-174

Description

• Header: a toggle button per replica (color, name and size, with dropped replicas marked), usable from the keyboard, and the time range picker shared with the classic page. useChartLegend's toggle now takes any click event, since it only reads the modifier keys.
• CPU: a line per replica, as a share of the replica's CPU allocation. When a reading passes 100%, the axis extends to fit it and a dashed line marks the allocation.
• Memory: a bar per replica as a share of the heap limit (RAM plus swap), on a fixed 0 to 100% axis so sizes with a lot of swap don't stretch it. The replicas' bars at one time overlap by half a bar, so each group reads as one point in time.
  • On 26.44 and later the bar splits into RAM and swap, and the legend gives the RAM and heap limits in bytes.
  • Earlier environments show the bar as heap alone, labelled "Heap".
  • Where no sample in the window has a heap limit, such as on the local emulator, the bars are a share of the memory allocation, and the labels say so. Only the views' rows on 26.44 and later carry that, so other windows keep the heap labels.
• Replica status: a row per replica, including one offline the whole window. Running, finished hydration and offline stretches, a marker for each out-of-memory kill, and a "Hydrating · N objects" badge at the now edge while a replica has objects left to hydrate.
• No readings: when no sample in the window has a reading, the card says why from replica status: replicas ran but weren't measured, every replica was offline after an out-of-memory kill (with the memory troubleshooting guide), every replica was offline with no recorded reason (with the hydration troubleshooting guide), or status is unknown. Cloud adds a contact-support link; self-managed points at metrics-server. Replica status always shows below.
• Markers: a dotted line where an index or materialized view was created on the cluster, labelled on the CPU chart. The objects come from useClusterObjectsLive, so DDL elsewhere in the environment doesn't redraw the charts. The cluster page mounts the objects collection bridge, as Object Explorer does.
• Replicas that share a name: a scheduled cluster gets a new replica id on every refresh, under the same name. Replicas of one cluster that share a name share one toggle, one color, one status row and one tooltip section. A past blue-green deployment's replicas keep their own, since they run alongside the current ones while those hydrate. A time's memory bars split only among the replicas reporting at that time, and bucket merging counts the same. A CPU reading with no neighbor, such as a short refresh merged into one bucket, draws as a dot.
• Processes: when a replica's size runs more than one process, a Processes button on the CPU chart opens a drawer with a tab per such replica; only the open tab reads. Each tab leads with the busiest process for CPU and for memory against the others, or says the load is even. Then one CPU chart and one memory chart draw every process as its own line, with a legend that names each process's line, highlights it on hover, and selects or clears any or all of them. A table (UniversalTable) lists each process's average and peak, computed from its samples. Readings are shares of one process's allocation, over the card's window up to 24h. The drawer reads each process's samples from mz_cluster_replica_metrics_history (a lookup on its replica_id index) through a SUBSCRIBE that is open only while the drawer is, starting at console: add data for the redesigned cluster page's resource usage #39507's rolling literal start.
• Tooltip: per replica, its status and readings at the hovered time. A Swap row appears only where the size has swap.
• Wide windows: neighboring buckets merge, keeping each metric's peak, so bars stay at least 10px wide. The time axis matches the classic graphs.
• Metrics tab: the system cluster info box, this card, then the existing resource intensive objects and freshness cards, unchanged.

Loading and deriving live in useResourceUsageData. The chart marks are memoized, so hovering only redraws the cursor and tooltip.

The classic page's check for a cluster reporting no metrics moves into hasNoUtilizationMetrics, which both pages now use. Its behavior is unchanged.

Differences from the classic graphs: the CPU axis extends past 100% instead of clamping, there's no disk chart, replica toggles replace the replica dropdown, and windows of 3h or less don't poll offline events.

Not yet checked in Safari or Firefox.

Verification

Verified using teleport to customer environments
ResourceUsage.test.tsx renders the card against MSW handlers and covers:

• each replica's CPU, memory and swap readings at a hovered time;
• heap alone on environments without the swap columns;
• memory labelled as a share of the allocation when no sample has a heap limit;
• the CPU axis extending past 100% with the allocation line, and staying at 100% otherwise;
• the out-of-memory marker in the tooltip;
• the limit labels with their byte sizes;
• a status row for a replica offline the whole window, and the status rows in a window with no usage;
• the count of objects a replica has left to hydrate;
• hiding replicas with the toggles;
• the creation marker label;
• the empty window and the load error;
• a week of a scheduled cluster's refreshes: one toggle and one status row for r1, a line per refresh with more than one point, and bars as wide as their own bucket;
• one tooltip section per name while hovering a refresh; a refresh that fits one merged bucket drawn as a dot; a past blue-green deployment's r1 kept apart from the current one;
• a Processes button only for sizes that run more than one process, and a drawer naming the busiest process, drawing a named line per process on each chart that the legend selects or clears, and listing every process's average and peak;
• status rows in the same order as the replica toggles;
• no fallback message while replica status is still loading.

processReadingsModel.test.ts covers each sample as a share of its process's allocation, memory against the heap limit with the allocation fallback, each process's peak per bucket, each process's average and peak from its samples (a process bursting one minute in twenty doesn't outrank a steady one over a day), and naming the busiest process against the others. The drawer test answers its SUBSCRIBE through a new mockSubscribeSocket helper, which sends a completed snapshot over the WebSocket.

ClusterMetrics.test.tsx covers the error shown when the clusters subscribe fails before its first snapshot.

ClusterDetail.test.tsx mocks the new Metrics tab.

🤖 Generated with Claude Code

[linear-code]

CNS-174

[def-]

QA LLM Review

1. MEDIUM -- Swap sizes get a "Disk" chart that plots swap

console/src/platform/clusters/ClusterPage/ResourceUsage/ResourceUsage.tsx:419

The disk chart and the tooltip's Disk row appear whenever diskPercent !== null, but on swap sizes (the M.1 sizes) disk_percent measures swap usage, not scratch disk. Any swap-size cluster that swapped during the window gets a "Disk: % of scratch disk in use (spilled state)" chart, and its tooltip shows the same swap bytes twice, once as "Swap" and once as "Disk". The PR says the opposite should happen: "Swap sizes run without a scratch disk and report none".

Details

• Swap replicas get no --scratch-directory, so clusterd reports no disk_bytes. The orchestrator then stores swap_bytes in disk_bytes (src/orchestrator-kubernetes/src/lib.rs:1616, "the Console expects the swap size to be reported in disk_bytes").
• For swap sizes, mz_cluster_replica_sizes.disk_bytes is the size's disk_limit, i.e. the swap budget. It is nonzero for M.1 sizes, since --heap-limit = memory_limit + disk_limit (src/controller/src/clusters.rs:824-833).
• So the utilization views' disk_percent (src/catalog/src/builtin/mz_internal.rs:8131) works out to swap used over swap budget. It is non-null whenever swap is above 0, because toReplicaUtilizationGraphData only nulls exact zeros.
• The classic page avoids this: it shows its Disk graph only when cluster.disk or replica.disk is set, and that flag is NOT swap_enabled AND disk_bytes != 0.
• The test fixture gives swap rows maxDiskPercent: null (ResourceUsage.test.tsx:105). "leaves out the disk chart when no replica reports disk usage" therefore passes on data that swap sizes never produce.

Fix: decide whether to show the chart and the tooltip's Disk row from the size's disk flag, the same check as the classic page's clusterHasDisk. Don't decide it from whether a disk reading exists.

2. LOW -- A transient null heap limit relabels the memory chart until the page remounts

console/src/platform/clusters/ClusterPage/ResourceUsage/useResourceUsageData.ts:156

hasHeapLimit reads a null heap_limit in a replica's latest metrics row as "this environment has no heap limit". In cloud, that row is also null whenever a metrics fetch fails, for example while a replica restarts or sits in crash-loop back-off. If every current replica's latest row is null when the page loads, the memory chart switches to "% of memory allocation" and "memory limit" and drops the byte sizes. It stays that way: useReplicaHeapLimits stops polling once each replica has any row, and it uses staleTime: Infinity.

Details

The orchestrator records an all-null row when the metrics API call fails (src/orchestrator-kubernetes/src/lib.rs:1577, :1590). It leaves heap_limit null when the clusterd usage fetch fails (:1572). mz_cluster_replica_metrics returns the latest row per process. A single-replica cluster that was just OOM-killed is the likeliest trigger, which is exactly when someone opens this chart. The bars still show the share of the heap limit for samples that had one, and on 26.44+ the orange line keeps its "RAM limit" label, so the chart contradicts itself. Keep polling while a replica's limit is null, or derive the flag from the history rows' heapLimitPercent.

[jdonelson]

Timelines are only built for replicas in graphData, i.e. ones that reported metrics in the window. A replica that's been offline the whole window has no samples, so it gets a toggle but no status row, though an offline stretch is exactly what the strip is for. If every replica is down, the card shows "No resource usage in this time period" and hides the strip entirely, which reads as idle, not broken. Status history already covers the current replicas, and transformReplicaTimeline already carries a pre-window offline transition across the window. Could the rows come from replicasById, and the empty state keep the status strip?

[leedqin]

Agreed. Rows now come from every replica, so one offline the whole window gets an offline row from status history, and a window with no usage keeps the status rows under the empty message. Both have tests.

[jdonelson]

Follow-up: my premise here was incomplete. In cloud a down replica still writes an all-null metrics row every minute (get_metrics falls back to ServiceProcessMetrics::default() when the metrics API call fails). So when every replica is down for the whole window, the card takes the hasNoUtilizationMetrics branch. It shows the metrics-collection alert and no Replica Status. That's the QA bot's Oct 8 finding.

A missing metrics-server produces the same all-null rows, so the readings alone can't tell the two apart. Status history can, and it's already loaded. Proposal, for when every reading in the window is empty:

• Some replica was online in the window: something was running and wasn't measured. "Your replicas were running, but no resource usage was recorded for this period." Then the deployment-specific advice below.
• Every replica was offline the whole window, and the last offline reason was oom-killed: "All replicas were offline for this period. They last went offline after running out of memory. See Troubleshoot memory usage."
• Every replica was offline the whole window, with no reason recorded: "All replicas were offline for this period, and Materialize didn't record why. If they keep restarting, see Troubleshoot hydration failures."
• Status unknown (history failed to load, or has no rows): "No resource usage was recorded for this period. The replicas may have been offline, or metrics may be unavailable." Then the deployment-specific advice below.

Deployment-specific advice, by appConfig.mode:

• Cloud: "If this continues, contact support." Customers can't install or fix metrics-server in cloud, so the current advice isn't actionable there.
• Self-managed: "Check that metrics-server is installed in your Kubernetes cluster."

In every case, show the Replica Status section below the message.

The memory and hydration guides each need a one-line entry in mz-doc-urls.json, as #36479 did. Support is already there. I'd avoid "increase the cluster size" for OOMs, since the memory guide lists several causes where a resize isn't the fix.

[leedqin]

Thanks for the follow-up. Implemented your proposal in 2488ee4: when no sample in the window has a reading, the card picks the message from replica status (ran unmeasured, all offline after an OOM, all offline with no reason, or unknown), adds the cloud or self-managed advice, and always shows Replica Status below. The two guides are in mz-doc-urls.json.

[leedqin]

Thanks. For 1, the Disk chart and the tooltip's Disk row are gone; the new page doesn't chart disk. For 2, the memory label now comes from the utilization rows: a window is labelled as a share of the memory allocation only when none of its samples has a heap limit. The heap limits query also keeps polling while a replica's latest limit is null, so the legend's byte sizes recover after a failed fetch.

[def-]

QA LLM Review

1. MEDIUM -- A cluster whose replicas are all down still loses its status strip

console/src/platform/clusters/ClusterPage/ResourceUsage/ResourceUsage.tsx:453

The new status rows assume an offline replica reports no samples, but in cloud a provisioned replica that is down still writes an all-null metrics row every minute. So when every replica has been down for the whole window, for example a single-replica cluster stuck Pending or crashing at boot, graphData is non-empty, hasNoUtilizationMetrics is true, and the card shows "Cluster metrics are unavailable ... ensure that pod metrics collection is enabled". The status strip, which would show the offline stretch, is never rendered. That is the all-down case the review thread asked to fix, and the alert points at metrics-server rather than the outage.

Details

Why the rows exist: the per-replica metrics task runs every 60s for as long as the replica is provisioned and sends whatever the orchestrator returns (src/controller/src/clusters.rs:963-967). When the pod has no running container, get_metrics returns ServiceProcessMetrics::default() (src/orchestrator-kubernetes/src/lib.rs:1577, :1590), and record_replica_metrics writes it as a row of nulls (src/controller/src/lib.rs:616-646). The _3h, _24h and _overview views keep those samples, giving null cpu_percent/memory_percent/disk_percent. Meanwhile a not-ready pod is recorded as Offline in status history (src/orchestrator-kubernetes/src/lib.rs:1386-1390), so the strip has the data it needs.

The new hasUsage === false branch only handles a window with no rows at all. The test fixture produces exactly that case: r2OfflineAllWindow drops r2's utilization rows (ResourceUsage.test.tsx:162, :179). Both new tests pass without covering what an offline replica actually reports. In multi-replica clusters the offline replica already had null samples and therefore a row, so the new no-samples path mostly covers the first minute after a replica is created.

To fix this, keep the status strip in the hasNoUtilizationMetrics branch, with the alert above it, or skip the alert when status history shows the replicas offline. A fixture that gives the offline replica null-valued rows would cover the real case.

[leedqin]

Thanks, confirmed and fixed in 6bc0626: the card now measures only its width with useParentSize, as UtilizationGraph does since #39611.

[leedqin]

Thanks. The all-down case is fixed in 2488ee4 along the lines of the review thread: empty readings no longer hide Replica Status. The memory chart now overlaps each time's replica bars by half a bar, with a surface ring between them, so each group reads as one point in time.

[jdonelson]

I think the current bar chart is struggling to display processes per replica. It worked well with 2 or 3 plots (1 per replica) but for hefty cluster with 2 replicas and 8 processes each it starts to lose visual fidelity. The colors and line styles will cycle and different processes will have the same color/style. And as the bars increase in number, we won't be able to fit them all in the bounds and the time window will have to shrink.

rendered example:

Some possible alternatives could be:

• Display only the max process size per replica.
• Display the max, min, and average using bounded range or candlestick graphs. Or the average could be a line while the max is shown via bars.
• Drill-downs or toggles so that we're not showing everything at once.

Here are illustrations of these ideas along with some others:
https://data.europa.eu/apps/data-visualisation-guide/differences-in-time-series

https://observablehq.com/@observablehq/whats-different-timeseries-forecast-performance?collection=@observablehq/analyzing-time-series-data
("Perceiving the vertical distance or the shortest path?" section)

[def-]

QA LLM Review

1. MEDIUM -- Clusters with SCHEDULE = ON REFRESH get blank CPU and memory charts on wide windows

console/src/platform/clusters/ClusterPage/ResourceUsage/ResourceUsage.tsx:367

The merge width (replicaCount: series.length) and the memory bar slots (ResourceUsageCharts.tsx:654) count every replica charted in the window, including past replicas that never ran at the same time. A scheduled cluster gets a new replica id on every refresh, so a 7-day window holds dozens of series that never overlap. Buckets then merge until each replica is a single point, so no CPU line is drawn and memory bars shrink below 1px. The status strip and the replica toggles also grow by one "r1" row per refresh.

Details

Each refresh turn-on allocates a fresh replica id (src/adapter/src/coord/cluster_controller.rs:737, reason OnRefresh at :1030). The utilization views keep dropped replicas through mz_cluster_replica_history, so graphData has one series per refresh, all named r1.

Probe: I rendered ResourceUsage at this head with a 7-day window, replication factor 0, and 28 dropped replicas (one every 6h, 3 hourly buckets each). 0 of the 28 CPU paths had more than one point. Every memory bar was 0.88px wide. There were 28 toggles and 28 status rows (672px). With an hourly refresh (160 replicas), bars were 0.036px wide and the status strip was 3840px tall.

Positive control: on the same fixture, with replicaCount: 1 and bars not divided by the series count, all 28 CPU lines draw and bars are 2.9 to 7.9px wide.

The 24h window has the same problem. An hourly refresh gives 24 series over 5-minute source buckets, so mergedBucketMs picks 8h groups, and each replica, up for about 10 minutes, merges into one point.

To fix this, size the merge from the largest number of series that have data in the same source bucket, and slot each time's bars only among the series present at that time. Dropped replicas that share a name could also share one status row and one toggle.

[leedqin]

Agreed, thanks. The card charts one series per replica again, and per-process detail moves to the workers drawer in #39889.

[leedqin]

Thanks, fixed in 0bb33c9: bucket merging and bar slots count only the replicas reporting at the same time, and replicas that share a name share one toggle, color and status row.

[leedqin]

Correction to my last note: per-process detail now lives in a Processes drawer on this card, opened from the CPU chart for multi-process sizes. It compares each replica's busiest process with the average one, in e0e0670.

[def-]

QA LLM Review

1. MEDIUM -- The tooltip lists every same-name replica at every hovered time

console/src/platform/clusters/ClusterPage/ResourceUsage/ResourceUsage.tsx:106

tooltipSectionsAt gives each series the state from its group's merged timeline, so a replica with no reading at the hovered time still gets a section. On a scheduled cluster, hovering inside any refresh lists one "r1 · Running" header per refresh in the window, and only one of them has readings. A week of hourly refreshes gives about 160 headers.

Details

On the PR's own scheduled-cluster fixture (27 refreshes, 7-day window), hovering inside refresh 10 produced a tooltip with 27 "r1 100cc Running" headers and one set of CPU/Memory/Swap rows. As a positive control, skipping members that have no point and aren't the group's representative brought it down to 2 headers.

Before this commit, each dropped replica's timeline covered only its own lifetime, so stateAt was undefined outside it and the section was skipped. A fix is one section per group at a time: take the member with a point, and fall back to the group's state only when no member has one.

2. MEDIUM -- A past blue-green deployment's replicas merge with the current ones and inherit their OOMs

console/src/platform/clusters/ClusterPage/ResourceUsage/resourceUsageModel.ts:320

groupReplicasByName assumes same-name replicas ran one after another because a cluster allows one live replica per name. But graphData spans the cluster's blue-green lineage (useClusterLineageIds), and the past deployment's r1 ran alongside the current r1 while the new deployment hydrated. Both get one toggle, one color and one status row. An OOM of the new replica is then shown on the old one too.

Details

Probe: I gave the current r1 (cluster u9) a lineage row for a past r1 (cluster u8), overlapping for 10 hours, and OOM-killed the new one inside the overlap. The card showed a single "r1 100cc" toggle and two CPU lines with the same stroke. At the OOM, both tooltip sections read "r1 100cc Out of Memory", including the past replica at 20% CPU that was never killed. mergeTimelines lets the later-starting segment win, so from the new replica's creation on, the row shows only the new deployment's states.

DataPoint carries no cluster id, so the grouping can't tell the two apart. Keying groups on the cluster as well as the name would keep a past deployment's replicas separate.

3. MEDIUM -- The scheduled-cluster test fails one hour in three

console/src/platform/clusters/ClusterPage/ResourceUsage/ResourceUsage.test.tsx:764

"charts a scheduled cluster's refreshes as one replica" depends on the wall clock. It fails whenever the window's first bucket falls on a UTC hour divisible by 3, which is 8 hours of every day, so it will flake on main and on unrelated console PRs.

Details

At the mocked 900px width the plot is 828px, which gives 4.9px per hour. With one bar at a time, mergedBucketMs therefore picks 3h groups, aligned to epoch multiples of 3h. Each refresh's three hourly buckets start at FIRST_BUCKET_MS + 6k hours. When that hour is a multiple of 3, all three buckets merge into one group, so every CPU path is a single M…Z point and lines is empty.

I ran it in a clean copy at 20:34Z and it failed, with all 27 paths single points. Shifting NOW_MS by 0 to 5 hours failed at 0h and 3h and passed at the other shifts.

Pinning the clock (vi.setSystemTime) makes the result deterministic. At the failing alignment, though, the chart really draws no CPU line for any refresh, so the assertion describes something the code doesn't guarantee for every window start.

[def-]

QA LLM Review

1. MEDIUM -- The Processes drawer's "average" is a mean of bucket peaks, so on wide windows it names the wrong busiest process

console/src/platform/clusters/ClusterPage/ResourceUsage/processReadingsModel.ts:108

summarizeProcesses averages each process's per-bucket peaks, not its samples. The headline ("Process N used the most CPU: X% on average, k× the other processes"), the busiest highlight and the table's "CPU average"/"Memory average" columns therefore inflate bursty processes. From the 3h window up they can name a mostly idle process as the busiest.

Details

The drawer buckets at max(windowMinutes * 1000, 60s) (ProcessesDrawer.tsx:169), so a bucket holds one sample at 1h but 24 at 24h. bucketProcessReadings keeps only each bucket's max (processReadingsModel.ts:69-70), and summarizeProcesses then takes mean over those maxima. The result depends on the window width, and the same data gives a different "busiest" process at 1h and at 24h.

Probe: I ran the PR's model with two processes, one-minute samples, and the drawer's bucket size. Process 0 bursts to 100% for one minute in every 20 and is idle otherwise, for a true mean of 5%. Process 1 holds a steady 40%.

• 1h window: averages 5% and 40%, and process 1 is busiest (correct).
• 24h window: averages 98.4% and 40%, and process 0 is busiest. The headline would read "Process 0 used the most CPU: 98.4% on average, 2.5× the other process (40.0%)", although process 0 used 8× less CPU.

Smaller variance differences have the same effect. A process that swings around a lower mean can outrank a steady process with a higher mean. The EVEN_RATIO "spread evenly" verdict compares the same inflated numbers.

The tests don't catch this, because every summary fixture has at most one sample per process per bucket. Fix: compute the summary's mean and peak from the raw samples in the window, or keep a sum and count per bucket next to the peak. The chart lines can keep their per-bucket peaks.

[leedqin]

Thanks, all three fixed in 9d00332. 1: the tooltip shows one section per name at a time, from the replica with a reading then. 2: replicas are grouped by cluster and name, so a past blue-green deployment's replicas keep their own toggle, color and status row. 3: the test no longer depends on the hour, and a refresh that fits in one merged bucket now draws as a dot instead of nothing, with a test at that alignment.

[leedqin]

Thanks, fixed in 9d00332: each process's average and peak now come from its samples rather than its bucket peaks, with a test where a process bursting one minute in twenty doesn't outrank a steady one over a day.