Repository navigation
[Sync] develop with upstream 0.24.1 (the headset rendering squash, IBL fallback, pack-loader LOD: #1359, #1375–#1387) - #82
Merged
Conversation
… and RealityKit's look in a mixed space (untoldengine#1359) * [Feature] Draw the XR frame through the compositor's rasterization rate maps (foveation) With foveation enabled in the layer configuration, visionOS hands each eye a rasterization rate map and textures of its physical size; the engine ignored the maps, so apps had to keep foveation off, which also forbids a render quality above the platform default. The stereo path now attaches the eye's map to every pass that rasterizes the scene at drawable size and sets the eye's screen-space viewport there; the full-screen passes keep copying physical pixels one to one. SSAO rebuilds view positions from a texel's screen position, and the occlusion culls (mesh, Gaussian chunk, per-splat) move their projected rects into the pyramid's physical layout through the map the pyramid was drawn with, so culling stays on under foveation. The Gaussian fragment takes its offset from the splat centre from a varying rather than from its position. On by default when the drawable provides maps; `.foveatedRendering(.disabled)` ignores them for diagnosis. Kernels rebuilt for every platform. * [Bugfix] Query the frame's drawables with queryDrawables on visionOS 26 A layer configured with foveation or a render quality aborts the app when the frame is queried with the single-drawable queryDrawable() ("BUG IN CLIENT: called cp_frame_query_drawable() which does not support setting render quality or fovea"). The frame now takes the drawable array on visionOS 26 (the old call on visionOS 2), presents every drawable, and draws the scene into a capture drawable only while its textures match the frame's resources; otherwise it is presented cleared rather than reallocating every sizeable resource twice a frame. * [Bugfix] Leave the render target size to the attachments while a rate map is attached Under a rasterization rate map Metal reads a descriptor's renderTargetWidth and renderTargetHeight, like the viewport, in screen pixels. The scene descriptors carried the physical texture size there, which clipped every foveated frame to the top-left physical-size corner of the screen: on the headset a screen-fixed rectangle, different in each eye, outside which the board vanished. The attached descriptors now leave the size at 0 (the attachments' own) and get the texture size back when the map is detached; the eye's viewport is taken from the map's screen size. * [Feature] Temporal anti-aliasing (.taa and .msaaTaa) The projection is jittered by a sub-pixel Halton offset before each frame's graph is built, and a resolve pass after the look pass blends the frame with the eye's history, reprojected through the depth buffer and the two frames' unjittered view-projections and clipped to the colour box of the pixel's neighbourhood. Sub-pixel detail a single sample flickers on — thin wires, fine normal maps, the periphery of a foveated frame where a texel spans several screen pixels — averages out over frames. Under a rasterization rate map the reprojection goes physical → screen → world → previous screen → previous physical through both frames' maps. Camera motion is handled; moving objects rely on the clip. Kernels rebuilt for every platform. * [Patch] TAA: variance clipping and a history weight that follows the rate map A pixel-thin feature — a wire, a line of text — is rasterized in some jittered frames and not others, and a min/max box of a frame that missed it clipped it out of the history, so it flickered at distance; the clip box is now the 3×3 mean ± clipGamma (1.25) standard deviations. Under foveation a spot the gaze has just moved onto has a history from the periphery, coarser than the new frame, which showed as a shadow fading over twenty frames; the history's weight is cut by the ratio of the two maps' local rates. Kernels rebuilt. * [Chores] Log the temporal resolve's parameters once * [Patch] TAA: keep depth-less pixels, Catmull-Rom history, moving-map and moving-camera tests A pixel at the clear depth — the background pass, a transparent quad drawn over passthrough — has no position to reproject from; fetched from the far plane its history ghosted or wiped it under a moving camera (the HUD pill and label flickered on and off). Such a pixel now keeps its own frame. The history is sampled with a Catmull-Rom filter (nine bilinear taps), which resampling every frame smears less than bilinear. The rate map on renderInfo.xrFoveation now also goes on the scene passes of the macOS path, so the foveated resolve is testable on a Mac: the new tests draw a still scene through a rate map whose dense zone jumps every frame, and move the camera, in both temporal modes; the residuals sit at the held-still floor. Kernels rebuilt. * [Bugfix] Light back faces with the normal turned towards the viewer The model pass draws without culling, so an open mesh shows its back faces — a dart's flight, a leaf — and those arrived with the front side's normal, pointing away from the viewer, and shaded as if unlit. A face whose normal points away from the viewer is now lit with it turned around. The test is geometric rather than [[front_facing]]: the pass sets no winding and assets arrive wound either way. Kernels rebuilt. * [Chores] TAA: a jitter scale for diagnosis (TAAParams.jitterScale) * [Bugfix] XR: a capture drawable does not advance the temporal history or the depth pyramid On visionOS 26 a frame carries a second drawable while the view is recorded or streamed. The engine drew the scene into it as into the built-in display, so each eye's temporal history and the occlusion pyramid were overwritten from the capture's pose on every frame: the display's next resolve read a history that alternated between its own pose and the capture's, and thin objects blinked while recording. A secondary drawable now resolves its frame as is and leaves the history, the matrices, the map and the pyramid to the built-in display. The frame's drawable count is logged when it changes. * [Bugfix] TAA: take the rate ratio on the axis where the history is coarsest A rate map is coarser in one axis than the other wherever its rows and columns differ, and Metal quantizes the requested rates, so a frame can be 0.6 texels per screen pixel in y where the history's frame was 0.25. The history weight was scaled by a ratio taken from the larger step of the two axes and judged such frames equal in rate, keeping a history 2.4× coarser at full weight: its blurred edge overrode the frame's and a thin object lost its edge rows on alternate frames (a headset HUD strip blinked at the one distance where it sat on such a zone). The ratio is now per axis, the smaller taken. The moving-map test now draws a map coarse in both axes and asserts, with the jitter off, that no lit pixel loses its colour to a history drawn through the other map. Kernels rebuilt. * [Feature] The XR optimisation set: pass skips, TAA into the history, GPU time per frame, the tile light stage, opaque geometry without discards Six branches of 2026-10-12/13 on top of foveation and specular anti-aliasing: skip the Gaussian pass on a frame without splat entities, skip the pre-composite pass on a passthrough frame it would only copy, TAA resolving into the history alone, a frame's GPU time beyond the previous frame's, the tile light stage with the multisample-count setting and the up-front tile-memory check, and opaque geometry through a model pipeline without discards. Kernels rebuilt for every platform. * [Chores] Shading-cost experiment switches: flat material, flat lighting Two switches, off by default, that take one kind of shading out of the combined pass and leave the rest of the frame as it is, so a measurement pass with one on gives the frame's cost without that shading. Flat material: the opaque draws go through the plain model pipeline compiled with the material shading out (the material's factors and the vertex normal, no texture sample, no derivative). Flat lighting: the light quad compiled to return the albedo untouched, no image-based light, shadows or lights; the tile stage stands aside. Both are function-constant specialisations of the existing shaders (`modelFunctionConstantFlatMaterial`, `lightFunctionConstantFlatShading`), selected through `setRendering(.flatMaterialExperiment)` / `.flatLightingExperiment`. Kernels rebuilt for every platform. * [Chores] Back-face culling experiment switch for the model pass The model pass draws both sides of every triangle, so a closed mesh sends its back faces through the tiler and the rasteriser too. As an experiment, `setRendering(.backFaceCullingExperiment(.counterClockwiseFront))` (or `.clockwiseFront`, for an asset wound the other way) culls the faces wound the other way in the G-buffer pass; the light quad is never culled, and off, the default, the pass draws as it always has. An open mesh loses its back with it on: the product version needs a per-material double-sided flag from the exporter. * [Feature] GPU time per pass, measured by the GPU itself `GPUPassTiming` carries a counter sample buffer through each frame's command buffer: every render pass writes a timestamp at the start and end of its vertex and fragment stages, every culling compute pass and the depth copy at their bounds, and the buffer is resolved when the command buffer completes. Each pass's time is averaged over the recent frames (`averages()`), the last frame is available as it came (`lastFrame()`), and the same label in one frame, one pass per eye, adds up. The GPU's clock is calibrated against the media clock, since its ticks are not documented. Off by default, `setRendering(.gpuPassTiming(.enabled))`; nothing is measured, and nothing changes, on a GPU without stage-boundary sampling. This is the per-pass view a Metal System Trace gives, on a device that cannot record one. * [Feature] Light the indirect part once per fragment, in the G-buffer Under MSAA the deferred light quad runs once per sample — a fragment function that reads the framebuffer is invoked per sample whether or not it declares a sample index — so every pixel's image-based light and cascade shadow were evaluated four times at 4×. The model fragment shader, which runs once per fragment, now evaluates them and carries them in the emissive attachment (the indirect light added to the emissive, the shadow factor in alpha), and the light pass adds them from there, keeping only the direct lights per sample. A function constant on each shader selects it (`modelFunctionConstantIndirectInGBuffer`, `lightFunctionConstantIndirectFromGBuffer`); the model and light pipelines follow `setRendering(.perFragmentIndirectLighting)` on the next frame, enabled by default. The picture is the same within the G-buffer's precision, and the tile light stage stands aside while it is on. Kernels rebuilt for every platform. * [Patch] Build the depth pyramid only for a frame whose depth could hide something Every frame copied its opaque depth and built the occlusion test's pyramid from it, whether or not the test would run on the next frame: with the test disabled (a camera step of zero) or a scene of one mesh entity, which hides nothing but itself, that was the frame's cost for nothing. The copy and the build now run only when the test is on and the culling set has at least `HZBOcclusionCulling.minimumEntities` mesh entities (two by default, `setRendering(.occlusionCullingMinimumEntities(_:))`) or the frame has splat entities, whose cull reads the pyramid; a frame without a pyramid marks it invalid, so no test reads a stale one. * [Patch] The temporal resolve presents the frame when the output pass would only copy it With temporal anti-aliasing the output transform pass read the resolve, encoded it for the display and copied the opaque depth for the compositor: a full-screen pass per eye to move the frame one texture over. The resolve now does that itself (`fragmentTAAPresentShader`): the display's colour texture is its second colour attachment, the display's depth its depth attachment, the encoding and the far-depth clamp are the output pass's, and the output pass stands down for the frame. Only when nothing is registered between the resolve and the output in the graph, the view is the lit one, and the present textures exist at the resolve's size; otherwise the frame runs as before. Kernels rebuilt for every platform. * [Feature] Light the opaque geometry whole once per fragment; the light quad stands down The combined G-buffer + light pass lit every multisample of every pixel through the light quad, reading the G-buffer from the tile: under 4× MSAA four evaluations of the direct light per pixel, and under a varying rasterization rate map (XR foveation) each of them at the map's multiplier. The indirect light moved to the geometry a commit ago; now the whole of the light does: the model fragment shader evaluates the sun through its cascade shadow, the point, spot and area lights with theirs, over the indirect light it already carries, and writes the lit colour to the attachment the quad would write. The quad then has nothing to do and the pass ends after the geometry — the forward way to shade, once per fragment, which under MSAA is once per pixel but at the edges. - `computeDirectLighting` (LightShader.metal, declared in ShadersUtils.h): the direct part of `lightTBDRSample`, factored out for the geometry to share; the quad's per-sample path calls it and is unchanged in what it computes. - `fragmentModelShader`: function constant `modelFunctionConstantLitInGBuffer` (`kModelLitInGBuffer`, index 5) binds the light blocks, the LTC tables and the point and spot shadow maps at new model-pass indices, and writes `GBufferOut.lit` ([[color(5)]]); the pipeline masks the attachment off when the constant is false. `kModelSunNeeded` keeps the sun's parameters bound for either per-fragment mode. - `renderInfo.perFragmentLighting` (on by default; needs `perFragmentIndirectLighting`, and stands aside for the flat-lighting experiment, which runs through the quad), `perFragmentLightingBuilt`, and `setRendering(.perFragmentLighting(.enabled | .disabled))`; the pipelines follow the setting on the next frame as the indirect one does. - The combined pass uploads the point, spot and area lights once, before the geometry, and binds them to whichever lights with them; with the geometry lit, the tile light stage stands aside and the quad's sub-pass returns after ending the encoder, on device and in the simulator. - Tests: PerFragmentLightingTests — the same picture as the quad at one sample and under 4× MSAA (a box over a slab under the sun, a point light and the image-based light), the frame one draw short with the quad gone, and the point light there. - All seven metallibs rebuilt. * [Feature] The forward pass holds the lit colour alone, and takes only what its fragments read With the geometry lighting itself per fragment, nothing read the five G-buffer attachments, yet the pass still carried them in its imageblock: 48 bytes per sample at 4× MSAA, which holds the GPU to 16×8 tiles. The opaque pass now has a second descriptor for that mode — the lit colour in the light pass's slot and the depth, 12 bytes per sample — and the model pipelines built for it declare that attachment alone: the shader's G-buffer outputs are compiled out. On Apple GPUs every interpolated float the vertex stage emits is written to the parameter buffer by the tiler and read back per tile, so the vertex-to- fragment layout is a vertex-stage cost in itself (measured on the board: the stage halves with the material's inputs gone). `VertexOutModel` carried the object-space position and normal, which the fragment stage transformed again, beside the world-space normal it also carried, and three members nothing ever wrote. It now carries the world position, the world normal, the tangent and the uv: twelve floats beside the position. - `GBufferOut` lives with its function constants in modelShader.metal; each member is a function-constant output: the G-buffer's by `kModelGBufferOut` (the geometry not lighting itself), the lit colour by `kModelLitInGBuffer`. - `renderInfo.forwardRenderPassDescriptor` (built beside the G-buffer's, foveated like it); `renderInfo.opaquePassDescriptor` picks the descriptor the pipelines were built for, which the combined pass and the two stand-alone model passes take. The combined pass labels itself "Forward", skips the G-buffer attachment setup and the tile stage in that mode. - `perFragmentLightingWanted` also stands aside for the G-buffer debug views, which read what the forward pass does not write. - `vertexModelShader` transforms the position to world space once; the model, transparency and gizmo fragment stages take it and the world normal as they are. - All seven metallibs rebuilt. * [Chores] Time the XR render loop's wait and submission phases in the engine stats The XR loop recorded its update phase and its whole period, which includes the wait for the compositor's optimal input time, so the CPU's own work per frame could not be read from the stats. `EngineTimingStats.xrWaitMs` is now that wait, `encodeMs` the submission phase (the drawables' command buffers encoded and committed), and `updateMs` + `encodeMs` the CPU's work. The loop also prints the compositor's frame timing once: optimal input time to rendering deadline, and to presentation. * [Chores] Say whether the engine was compiled optimized An app's Debug configuration builds the engine package at -Onone, and a frame that fits comfortably with the engine optimized is then bound by the CPU encoding it: on a headset the opaque draws took five times longer to encode and the GPU downclocked while it waited, so a measurement timed the compiler. `EngineBuild.isOptimized` reads the engine module's own assert configuration, never inlined, for a measurement to print it. * [Feature] The sun shadow's filter takes 16, 8 or 4 taps, as the softness settings say The cascade shadow filtered every fragment with the whole 16-point Poisson disk, a hardware-compared sample each: on the board under a foveal map that was a fifth of the whole opaque fragment stage (Mac, release: 0.63 ms with 16 taps, 0.52 with 8, 0.49 with 4). `ShadowSoftnessSettings.filterTaps` (16 by default, else 8 or 4) now picks every second or fourth point of the disk, which keeps its spread with fewer compares and leaves a grainier penumbra that a temporal resolve smooths. On a headset `xrFilterTaps` applies, 8 by default: the frame is drawn twice at a high resolution under a temporal resolve, and the whole disk cost 1.5 ms of an 11 ms frame there (judged acceptable at 8 on the board). - `CSMUniforms.shadowFilterTaps`, in the Metal struct and its Swift mirror, carried by `ShadowSystem.makeUniforms`. - Tests: the headset takes 8 taps by default and the Mac 16; a count asked for is the disk or every second or fourth point of it. - `sampleCSMCascade` runs one of three loops with a constant trip count, which the compiler unrolls: a single loop over a count read from the uniforms ran a quarter slower at 16. * [Patch] The temporal resolve reads the maps' local rates from derivatives and the history with five taps Under foveation the resolve decoded each rasterization rate map twice per pixel: once for the pixel's position and once a pixel over, for the map's local rate, which weights a history coarser than the frame. The second decode of each map is gone: the mapping is now computed for every pixel before any pixel leaves, and the rates are the derivatives of it across the 2×2 quad — this frame's screen position per texel, the history's texel per screen pixel. The history's Catmull-Rom filter takes the five taps that carry its weight instead of nine; the four corner taps weighed a few thousandths at most, and the five are renormalised. - Same picture on the Mac within the temporal tests' tolerances (the still scene's settling test fails as it did before this change, at 0.00428 against 0.004; it failed at that figure on the branch's integration commit). - All seven metallibs rebuilt. * [Patch] The look pass presents the frame when the output pass would only copy it Without a pass after the look (no anti-aliasing pass after it, nothing registered before the output) the output transform read the look output, encoded it for the display and copied the opaque depth: a full-screen pass per eye to move the frame one texture over, 0.9 ms on a headset. The look pass now presents the frame itself in that case, as the temporal resolve does: the display's colour texture is its second attachment, the display's depth its depth attachment, the encoding and the far-depth clamp the output pass's, and the output pass stands down for the frame. - `lookColor` (LookShader.metal) is the look's colour as a function; two fragment functions share it: `fragmentLookShader` writes it alone, `fragmentLookPresentShader` writes it, the encoded frame and the depth. - Pipeline `.lookPresent`; `renderInfo.lookMayPresent`, set by the graph when the look is the last pass before the output; `renderInfo.lookPresentedThisFrame`. - Tests: LookPresentFoldTests — in MSAA mode the look presents, the output pass is absent from the frame and the drawable holds the encoded look output; with FXAA after the look the output pass presents as before. - All seven metallibs rebuilt. * [Bugfix] The temporal resolve gives a history with no surface in it no weight Over a transparent background — a headset's passthrough — a surface that moves across the frame covers pixels whose history holds nothing but the cleared background: transparent, and black. The resolve blended that history in at full weight, and black sits inside a black-and-white surface's colour box, so the variance clip let it through: the surface's leading edge faded toward the room behind it for the twenty frames the history takes to catch up, and moving in or out faded a ring all around it (seen on a dartboard against a wall: a dark smear ahead of any lateral move, a dark halo when walking up to it). The history's weight is now scaled by its coverage against the frame's, whole again from a quarter of the frame's coverage up, so an edge the frame covers a little more of keeps its history. - Test: TAATransparentHistoryTests — a box moving forty pixels a frame over a transparent background (the passthrough's immersion style) keeps its leading edge opaque and as bright as its middle. - The temporal suite is unchanged in what it accepts: the moving-camera and still-scene tests pass with the change (the still scene settles closer than before: its edges no longer take a history that covers them less). - All seven metallibs rebuilt. * [Patch] The room's estimated light lights an object as bright as the passthrough shows the room The environment probe's camera scale reference was divided by a white point of 5000 and the result capped at 8, which lit a dartboard on a headset at about a fifth of the brightness RealityKit gives it from the same probe in the same room: its white segments read 13 of 255 against RealityKit's 61, with the passthrough wall the same in both. Calibrated against that comparison, the match is the reference over 1000 (a reference of 19016 wants a scale of about 19), and the cap is lifted to 64 to let it through. * [Feature] The opaque pass writes each pixel's motion, and the temporal resolve follows it The resolve found a pixel's history by reprojecting its position with the camera's motion alone, as if nothing in the scene moved on its own. A surface that does — a board carried across the room, a thrown dart — had for history itself shifted by a frame of its own motion: a black-and-cream pattern blended over that shift averages to a muddy brown, a metal number picks up the brown around it, and only a motion straight towards the viewer, which scales a radial pattern about its own centre, survived. In its forward mode the opaque pass now writes each pixel's motion since the previous frame (NDC, this frame minus the previous), from the entity's previous world transform and the eye's previous view-projection, and the resolve reads the history where the surface was. - `ObjectMotion` keeps each entity's world transform of the previous frame (once per frame; both eyes of a headset frame share it). - `Uniforms` carries the unjittered view-projection, the previous one and the previous model matrix; `VertexOutModel` the vertex's current and previous clip positions; `GBufferOut.velocity` is the forward pass's attachment 6 (rg16Float, multisampled and resolved to `velocityMap`). - The resolve takes the velocity (`taaPassVelocityTextureIndex`, `TAAConstants.velocityValid`) over the camera reprojection when the pass wrote it; the deferred path keeps the camera's. - The resolve's "presented this frame" is reset as each frame begins, not only when it resolves: a test order could leave it set in a non-temporal frame. - Test: TAAMovingObjectTests — a box moving under a still camera resolves as close to its own frame as a resolve held still. - All seven metallibs rebuilt. * [Feature] A tone map of none, for a mixed-immersion space to show the scene's linear colour The look pass always ran the scene through a filmic curve (ACES or AgX). In a mixed-immersion space the passthrough sets the display's response and the compositor applies it to the scene as well, so the curve compressed an object's whites and lifted its blacks against the room: a dartboard's cream segments read 87 of 255 against RealityKit's 113 for the same object in the same room, its blacks 15 against 9. `TonemapOperator.none` leaves the colour as it is (clamped at zero); `setPostFX(.tonemapOperator(.none))`. * [Bugfix] The culling's compute reads the list and the frustum of its own frame The frustum cull wrote the frame's entity list and frustum planes into the frame's slot of their rings and dispatched the GPU on the slot before: the frame before's list and planes, under this frame's count. When the list had just shrunk (a small part dropping out as an object recedes or turns), the entities at the end of the old list were never tested and vanished for that frame; the frustum was a frame late on every camera turn. On the headset the dartboard's face went black for a frame whenever the board moved away or sideways (the case behind it showed through), never as it came closer, when the list only grows. Both culling paths now bind the buffers they wrote, and the ring's read accessor returns the frame's own slot: the ring keeps the frames in flight off each other's slots, it does not stagger a frame behind its own data. Tests: an entity after one that left the list is still tested in the frame the list changed; the frame the camera turns in sees what it turned to. Both fail on the old code. The board's tour on the Mac had seven dark frames a loop before, none after. * [Chores] XR lighting diagnostics name the probe's format and count its publishes The diagnostics gain the probe texture as ARKit hands it over (pixel format, face size, mip count), the intensity scale as published (smoothed) and the number of prefiltered probes published, for the viewer's summary line. * [Bugfix] The room probe's faces light the sides of the world they stand for The headset's room probe is a cube map in the world's frame. The maps the renderer reads are written from each texel's direction in the pre-filter's own frame (z the pole, the azimuth from x and y) and read back with a world direction through the equirect lookup (y up, the azimuth from x and z); the two compose to a fixed turn of the axes, and the cube was sampled with the pre-filter's direction. So the probe's +y face, the ceiling, lit the left of a sphere, what stood behind the wearer lit its underside, and a chrome ball reflected a side wall for the room behind them. The static maps got away with it, since an equirect image reads the same whichever way its azimuth is turned. The cube is now sampled with the texel's world direction. Tests light a white sphere from one face of a probe at a time: the ceiling lights its top, the face behind the camera the side facing it, the right face its right. Kernels rebuilt (they also carry the look pass's display primaries, which follow). * [Feature] The look pass can show the scene with the display's P3 primaries, as RealityKit does RealityKit shows a mixed-immersion scene with the panel's wider primaries: measured on a panel of swatches both renderers built from code, its colours are the Display P3 → sRGB matrix of the sRGB values they were given (a red of (0.72, 0.13, 0.11) shows as (192, 0, 8) of 255, a cream of (0.93, 0.85, 0.62) as (231, 207, 145)), its greys unchanged, and its default tone map the identity below white. Shown next to it, the engine's faithful sRGB reads as faded. setPostFX(.colorPrimaries(.displayP3)) applies that matrix in the look pass after the tone map and the grade: every saturated colour widens, and a channel the matrix takes negative is outside sRGB and clips to black, as it does there. The default, .sRGB, keeps the colour as it is. Tests cover the setting, and the red above through both choices. * [Chores] XR lighting diagnostics report the probe's face luminance The mean luminance of each of the probe's six faces, from the tint readback the prefilter already makes: where the room's light is in the probe's frame, to compare with the side it lights. * [Feature] A render path setting names the deferred, hybrid and forward ways to shade The two per-fragment settings were named for the mechanism each moves: the indirect light into the geometry pass, then the direct light after it. What they make together had no name. setRendering(.renderPath(_)) gives it one: deferred (the G-buffer and the light pass), hybrid (the indirect light per fragment, the light pass for the direct), forward (the geometry lights itself whole and the quad stands down). getRenderPath() reads back the path the two settings make; the direct light per fragment needs the indirect, so that one alone reads as deferred, as the renderer treats it. Forward stays the default. * [Bugfix] The newest room probe waits for the prefilter in flight instead of being dropped ARKit sends its environment probe estimates in a burst when the provider starts and may then send nothing for a long while in a still room. The second estimate of the burst arrived while the first was still being prefiltered and was skipped, so the session kept the first, coarser estimate: a board pass showed two updates accepted, one published, nothing for the next thirty seconds. The probe system now keeps the newest probe pending: it is prefiltered when the one in flight finishes, or when the rest of the minimum interval has passed (a timer takes over the gate that shouldAcceptProbeUpdate used to apply by timestamp), and a newer probe supersedes an older one still waiting. Disabling the system drops the pending probe. On the headset a still session now publishes both estimates of the burst and a moving one every estimate. * [Docs] Why RealityKit shows a mixed-immersion scene with P3 primaries, and what sunlight does to it The colour management guide names the system's colour pipeline (Display P3, per WWDC24 session 10092) as the reason an sRGB-format layer keeps its colour while RealityKit widens it, and notes that RealityKit's own adaptation in bright sunlight (part transparent, a curve, darker surfaces) is the system's and not a target. * [Patch] The room light fades from one probe estimate to the next instead of switching The published scale and tint were smoothed, but the prefiltered maps swapped the frame a new estimate arrived, so the light's direction and colour changed at once: a turn towards a window, or a train passing a building, read as a bug. Each estimate is now read back (its first mip, six faces, and its last mip for the per-face light and the tint) and becomes the target. What is shown moves towards it exponentially, texels, scale and tint together, with lightingSmoothingTimeConstant (3 s now): every transitionStepInterval (0.1 s) the blended cube is uploaded, its mips regenerated, prefiltered and published, about a millisecond of GPU a step, until it reaches the target. A newer estimate replaces the target mid-way. The first estimate of a session is shown as it is. finishPrefilter publishes what it is given and says whether it did; the pending-probe queue now feeds the readback. * [Chores] Retire the shading-cost and back-face culling experiment switches The flat-material and flat-lighting pipelines and the back-face culling switch were measurement instruments for the headset work: each took one piece of shading out of the frame so its cost could be read. The measurements are done (the forward path and the per-fragment indirect light came out of them), and the switches were API surface, two extra pipelines and function constants for nothing a game would set. Gone: setRendering(.flatMaterialExperiment / .flatLightingExperiment / .backFaceCullingExperiment), the modelFlat and lightFlat pipelines, the two function constants (the others keep their indices), the renderInfo fields, and the tests that exercised them. The model pass no longer sets a cull mode (the default, none, is what the switch's off setting set), and the light quad takes the light pipeline directly. * [Chores] Two render tests pin the state they read The colour primaries test reads the look pass's output of an emissive swatch; a colour grade left enabled by an earlier test in the suite changed what it read. It now switches the grade off for its run and restores it. The GPU pass timing test looked for the "G-Buffer" pass; with the forward path as the default the combined pass is "Forward", so it accepts either. * [Bugfix] The room probe is convolved, not sampled: no ridges on a white sphere The headset's probe is eight texels a face. Its irradiance and specular maps were built by importance sampling it, 128 samples a texel of the maps, and the maps kept the structure of the sample set: with a window's worth of light in one texel, whether none, one or two of the samples landed in it turned with the normal, and a white sphere under room light showed ridges and spirals where RealityKit shows a gradient. The maps are convolved now. A kernel projects the probe onto its first nine spherical harmonics over a fixed set of 1024 directions (one threadgroup, reduced across it) and the irradiance pass evaluates them with the cosine lobe's band weights; light this smooth is held exactly. The rougher specular mips (roughness 0.25 and up) take the same fixed directions weighted by the GGX lobe about the normal; the sharper mips keep importance sampling, whose lobe is narrower than a texel. The harmonics live in a buffer of the lighting texture set, projected and read in the same command buffer as the prefilter. The static path, which reads large equirect images, is unchanged. Kernels rebuilt. XRProbeOrientationTests pass with the convolution, and a sun texel on a white rough sphere renders a smooth gradient. * [Chores] The batching tests give the scene channels back their render mode One of them hides the context geometry channel to check what a streamed tile hides, and the mode is global: every entity of every render test after the class drew nothing (a moving box "not in the frame", a lit sphere reading zero on all sides). The class's teardown now sets the context and ghost channels back to normal. * [Docs] The Vision Pro rendering profile, as measured What a mixed-immersion frame needs to look like RealityKit's, and what the headset hands the engine, kept with the numbers: the settings and why, how RealityKit shows a scene (no curve below white, P3 primaries, its own adaptation in sunlight), the room probe (an eight-texel cube with a scale reference, estimates in a burst, the convolution and the prefilter's frame), how to measure (captures, the calibration panel, a pass), what the device does with the dartboard, why motion judders, and the pitfalls met on the way. Linked from the README, the docs index and the XR immersion guide. * [Docs] The profile page says what a Mac rendering for the headset keeps and what it may raise A remote immersive space keeps the presentation half of the profile (the frame is still composited on the headset next to the passthrough) and takes the Mac's budget for the rest; the cadence and the link's latency are what the Mac does not buy back. * [Chores] Format five files as the pinned SwiftFormat (0.60.1) wants them The continuous-integration lint runs SwiftFormat 0.60.1; a newer local one reported other rules and missed these: single-line property bodies wrapped, a redundant nil default, three redundant selfs, an indent, a blank line before a declaration. Formatting only. * [Bugfix] The pass timing's frame crosses into the completed handler as a Sendable value Under strict concurrency the frame captured by the command buffer's completed handler was a non-Sendable value (it holds the Metal sample buffer), the one warning site the Swift 6 guardrails count against a budget of none. The frame is marked unchecked Sendable: the handler reads its sample buffer only once the GPU has finished writing it, and the slot's in-flight flag keeps the encoder away from it until then. * [Chores] The renderer tests step gets 60 minutes The step ran out at 45 minutes with 47 of 1,286 tests still to go: on develop it takes 38.5 minutes, and the seventeen render test classes of the headset work (the temporal resolve, the tile light stage, foveation, the probe, the forward path) push it past the limit on the runner. Sixty minutes leaves the headroom the numbers ask for. * [Bugfix] No tile light stage on a paravirtualized GPU, and two tests stand on their own The continuous-integration runner's Metal device is "Apple Paravirtual device": it reports Apple family 4, compiles the tile light stage's pipeline, and renders the stage black, so the deferred path lit nothing there while the quad and the forward path were fine. The stage gets no pipeline on such a device, which lights through the quad and lets the tile stage test skip itself. Two tests passed only on state an earlier case left behind, which the runner's parallel workers do not provide: the pyramid test placed no mesh, and the depth pyramid is built only for a frame whose cull found at least two, so it now places two boxes of its own; the temporal settling bound of 0.004 sat on the measured 0.0042 of a fresh process (0.0030 came from a warm one), and is 0.005, an order of magnitude under a drift or a smear. * [Bugfix] The render path defaults to the platform's: forward on visionOS and in the XR loop, deferred elsewhere From the review of untoldengine#1359: the two per-fragment settings defaulted to on, which made forward the default on every platform, while only the headset had been measured. `RenderInfo.defaultRenderPath` is forward on visionOS and deferred on every other platform, so macOS and iOS run the deferred renderer as develop does; the two settings take their defaults from it. The XR loop adopts forward as it initialises unless the game chose a path (`renderInfo.renderPathChosen`, set by `.renderPath` and by either toggle; `adoptXRRenderPathUnlessChosen`), so a Mac rendering a remote immersive space gets the headset's path too. RenderPathTests checks the platform's default and the adoption; the profile and architecture docs say where each path is the default. * [Chores] The hybrid path is drawn by name, and the render tests pin their path and restore it `.hybrid` was named only in RenderPathTests, which checks that the two settings compose it; no test drew it. HybridRenderPathTests does: the same picture as deferred at one sample and under 4× MSAA, the same picture as forward, the light quad's draw back (forward's draws + 1) with the tile light stage standing aside, and the point light's contribution — the frame with it less the frame without — equal to the deferred path's, so a direct light the geometry added as well would show as twice it. Measured on an M4 Max: hybrid against deferred mean 0.00004 and max 0.002 per channel, against forward mean 0.00003 and max 0.001; the point light's contribution differs by 0.0005 at most. The two existing per-fragment classes pin their paths with `.renderPath` (deferred and hybrid, hybrid and forward) and, with TileLightStageTests and XRProbeOrientationTests, restore the path they found instead of "enabled", so none leaves a path to the next test in the same worker now that the platform's default is deferred. * [Chores] The moving box under TAA on the deferred and hybrid paths, and the docs say which path writes motion The deferred and hybrid paths write no motion vectors, so under TAA an object moving on its own relies on the neighbourhood clip there. TAAMovingObjectTests runs its box on the three paths at two strides: 0.16 a frame, most of the box's width, where the clip alone rejects the stale history, and 0.02 a frame, a few pixels, the ghosting case. Measured on an M4 Max (a plain box over a plain background): the resolve's excess over a resolve held still is 0.1–0.3 % of the box's motion with the motion written and 0.2–0.4 % without, under `.taa` and `.msaaTaa`; the test bounds all three paths at a tenth of the motion and prints the figures. The architecture doc's TAA paragraph said there were no motion vectors yet; it says now that the forward pass writes them and the other paths rely on the clip, as do the `.taa` and TemporalAntiAliasing.swift comments. * [Chores] The clip positions the vertex hands the velocity are named by their components `VertexOutModel.currentClip` and `prevClip` hold the clip position's x, y and w, not xyz: they are `currentClipXYW` and `prevClipXYW`, and `modelVelocity` names the w it divides by. The seven kernel metallibs are rebuilt. * [Patch] The render test setup resets the immersion style, which the graph builder's immersion-mode case left at AR RenderGraphBuilderTest's testSupportedRenderExtensionStagesResolveAcrossImmersionModes tries .none, .mixed, .full and .ar in turn and left renderInfo.immersionStyle at the last one. Nothing reset it: resetGlobalEngineState did not touch it and the renderer created for the next test keeps it, so every later test in the same process drew the AR base pass. In a serial run the temporal rate-map test failed behind that class (lit pixels lost to a history drawn through the other map); CI never saw it because it runs each render test method in its own process. The base setup now resets the style to .none with the other per-test globals, and the case restores what it found. * [Chores] The kernel source hash records this branch's shaders Develop's stale-kernel check (untoldengine#1357) hashes the Metal sources into .metalsource.sha256 and warns at build time when the shipped metallibs may be behind them. Rebuilding the kernels with buildkernels.sh leaves the seven metallibs byte-identical, so they already match the shaders; the hash file now says so. * [Bugfix] The back-face turn decides on the face, not on the interpolated normal The model pass turns the normal of a face that points away from the viewer, so an open mesh seen from behind is lit. It decided on the interpolated vertex normal, which on a kit whose normals are averaged across hard edges (the Simple Dungeons walls and balusters carry corner normals 45–72° off their faces) passes 90° from the viewer before a front face's far edge: that strip of every such face was lit with its normal turned into the object, black under a room's light. Seen by Harold on a Vision Pro as dark bands along every model edge. The decision is now made on the face: the cross product of the world position's screen derivatives, oriented by the vertex normal, as the transparency pass already does. A face within a few degrees of edge-on, or one whose vertex normal lies within 12° of its plane, is left as it is. The seven metallibs are rebuilt. BackFaceFlipTests draws a quad under an image light that comes from a cap around its face normal, seen from above at a grazing angle on the forward path: with its normals averaged outwards it is lit all over (the interpolated-normal decision left 12 % of its pixels dark), and with its normals turned away it is lit all over too. --------- Co-authored-by: Javier Segura <javier@miolabs.com>
* [Release] Preparing release 0.24.0 * [Chores] update changelog
Fixes untoldengine#1373. Resolves the 8 categories of compiler warnings flagged in UntoldEngineTests and UntoldEngineRenderTests: #file/#filePath mismatches, unused/never-mutated locals, two unreachable dead-code branches, a redundant Equatable conformance, main-actor isolation on tearDown overrides, Sendable closure captures, a useless nil-coalescing, and deprecated Material init usage. Verified a clean swift build --build-tests produces zero warnings, and both UntoldEngineTests (1727 tests) and make testrenderer (1328 tests) pass with no regressions.
Reshape Tests/UntoldEngineRenderTests/Resources/ to be GameData-shaped
(untoldengine create layout) so UntoldEditor recognizes it as a project
and UnitTestRender.untoldscene/Models can be browsed and edited there
instead of hand-editing JSON.
- Move Scenes, Models, Animations, Materials under a new Resources/GameData/
directory; fix the Guassians -> Gaussians typo along the way. Materials
moves too even though it's not in ProjectLayout.gameDataCategories,
since LoadingSystem resolves it relative to assetBasePath and the editor
itself expects it alongside GameData's other categories.
- Add a minimal Resources/project.yml placeholder: UntoldEditor's
openProject() gates on <name>.xcodeproj or project.yml existing before
it even looks for GameData.
- Update DemoUtils.demoResourcesURL() and two hardcoded CLI/engine test
fixture paths to the new GameData-relative locations (they read straight
off disk, not through a bundled resource lookup).
- Leave BaseRenderSetup's assetBasePath pointed at the bundle root:
SwiftPM's .process("Resources") flattens the whole tree regardless of
GameData nesting, and the existing flat-bundle fallback in
getResourceURL already handles that.
…ine#1372) (untoldengine#1381) Adds testGlassTarget() to RendererTests, wired into the existing PSNR pipeline the same way testTransparencyTarget() is, with a glass/transmission scene added to the shared UnitTestRender.untoldscene fixture. Also fixes a deadlock this surfaced: BaseRenderSetup's default initializeAssets() blocked on XCTestExpectation.wait(for:timeout:) for loadUntoldScene()'s completion, but that completion always arrives via a Task { @mainactor in ... } hop (setEntityMeshAsync / deserializeScene's AsyncLoadTracker). A thread-blocking wait() never lets that hop run, so it only "succeeded" before because UnitTestRender had zero `.model`-kind entities (fully synchronous/procedural, no Task hop to wait on). The glass panes are real `.model` loads, so every test relying on the default initializeAssets() (~37 classes) started deadlocking once they were added. Fixed by making initializeAssets() kick the load off without blocking, and awaiting completion in setUp() via the existing (already async-safe) waitForOutstandingAssetLoadsToFinish() drain before setVisibleEntities() runs. Regenerates the reference images affected by the scene change (glass panes visible in frame) for RendererTests and PostFXTests.
…ntoldengine#1382) Zero-fill IBL textures when generateHDR fails, instead of leaving uninitialized/garbage data bound to the render pass, so a failed bake is ignored rather than turning models black. Also skip the bake attempt entirely when hdrURL is empty (no environment configured), which stops the spurious iBLCreationFailed error previously logged on unrelated viewport resizes after loading an HDR-less scene. Fixes untoldengine#1380
…ngine#1383) Adds CoolBasket, CoolBowling, ImmersiveGol, CoolMirror, and CoolFutbol, which were present in UntoldArcade's README but missing here.
… (untoldengine#1385) Reverts commit 5cf22298861a90edc9eb070dd384b1cf09a5d170f (untoldengine#1357). The build-tool plugin triggers Xcode's malicious-code warning, requiring manual trust approval, which is disruptive enough to warrant removing it. A different mechanism will be needed to catch stale .metallib builds. Closes untoldengine#1384
…1386) Saving and reloading a scene used to drop three things the pack loader sets up at first load: shared GPU builds for repeated placements of a model, the LOD chain of a multi-node model (so its parts switch level together), and LODComponent.selectsByScreenSize/screenSizeRadius (never serialized). Reload went through addLODLevel and the plain mesh path instead of the pack loader. - LODData now serializes selectsByScreenSize, screenSizeRadius, and levelsShareMaterials. - deserializeScene shares one UntoldBuildCache across the whole reload, so repeated placements of the same .untold build once. - A new packLODChain(forModelAt:) re-derives a multi-node model's LOD chain on reload by finding the .untoldpack that placed it, since a saved scene has no live link back to the originating pack. - Also fixes a related pre-existing bug surfaced while testing this against a real scene: LOD level reloads fired as unbounded concurrent Tasks instead of going through the same bounded dispatcher base-mesh loads use, which could hang scenes with many LOD entities on reload. Fixes untoldengine#1326
…headset rendering squash, the IBL fallback and pack-loader LOD fixes (untoldengine#1359, untoldengine#1375–untoldengine#1387) Upstream squash-merged the headset rendering branch (untoldengine#1359) whose commits the fork had already merged as a branch (through e7118fa), so the same content arrived twice: in the 15 conflicted files that upstream does not change beyond that branch head, the fork's versions stand (they carry the later shader-cost cuts, the instanced pipelines and the frame-time loop). Three files take upstream's additions on top: modelShader.metal decides a turned-away face from the position's screen derivatives (untoldengine#1359's last review fix) under the fork's sun and shadow branches; RenderInitializer keeps the instanced pipeline refresh and gains the IBL zero-fill (untoldengine#1382); UntoldEngine.swift reports 0.24.1. The .metalsource.sha256 goes with the stale-kernel check upstream reverted (untoldengine#1385), and the seven Metal libraries are rebuilt from the merged shaders with buildkernels.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the fork's
developup to upstreamdevelopat93bc8068(release 0.24.1).What upstream brings (10 commits since the last sync, #78)
modelShader.metal.metalsource.sha256goes with it)How the conflicts were resolved
The fork had already merged the untoldengine#1359 branch commit by commit (through
e7118fa8, #77–#79) and built on it: thexr_shader_costcuts, the instanced pipelines, the frame-time loop. Upstream's squash re-delivered the same content, so git saw both sides change the same lines in 25 files.git diff e7118fa8 upstream/develop -- <file>is empty): the fork's version stands.modelShader.metal: the fork's sun and shadow branches stay; the back-face test takes upstream's derivative-based decision.RenderInitializer.swift: the fork's instanced pipeline refresh stays; upstream'szeroFillIBLTextureand the no-HDR guard come in.UntoldEngine.swift: the fork'supdateXR(presentationTime:)stays; the start-up line reports 0.24.1.buildkernels.sh.Checks
swift build --build-tests: clean.UntoldEngineTests: 1751 passed. The one failure,ExternalRenderExtensionPackageTests, is the checkout's folder name: the fixture package depends on the engine by relative path and SwiftPM names a path dependency after its folder, so in a worktree calledUntoldEngine-sync1027the fixture'sUntoldEngineproduct lookup fails ("unknown package 'UntoldEngine' ... valid packages are: 'UntoldEngine-sync1027'"). It passes in a checkout namedUntoldEngine, which is what CI has.UntoldEngineRenderTests, the suites the merge touches (RenderGraphBuilder, StaticBatching, GlassShading, Transparency, HZB gate, FrameTimeLoop, InstancedDraw, Renderer, AssetLoadingGate, BlendedSubmeshFlag, MaterialAlphaDecoding): 249 passed.swiftformat --linton the two hand-merged Swift files reports the same tenwrapIfStatementBodiesand twowrapPropertyBodiesfindings as the fork's current versions (swiftformat 0.62.1 here against CI's 0.60.1), nothing new.