Skip to content

gloas: MEV-Boost/commit-boost docs, sidecar AuthData warning, masked builder URLs, not_published outcome - #3152

Merged
iurii-ssv merged 6 commits into
stagefrom
c/ssv-glamsterdam-mev-boost-affcd6
Oct 28, 2026
Merged

iurii-ssv merged 6 commits into
stagefrom
c/ssv-glamsterdam-mev-boost-affcd6

Conversation

@iurii-ssv

@iurii-ssv iurii-ssv commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Why

After Gloas, the beacon node contacts each configured builder itself over the produceBlockV4 POST (beacon-APIs#630), so SSV doesn't need MEV-Boost or commit-boost. Operators may still run one: commit-boost has merged its Gloas builder-API endpoints (Commit-Boost/commit-boost-client#505), and MEV-Boost's are in review (flashbots/mev-boost#914). Both forward each bid request to the one relay named by the auth data it carries, so each needs one SSV builder entry per relay, with URL set to the sidecar and AuthData set to the relay's identity.

docs/EXTERNAL_BUILDERS.md didn't say any of this, and its pre-Gloas sections didn't say they stop applying at the fork. The config also has a silent trap: an entry that points at a sidecar without AuthData defaults to the sidecar's own hostname (localhost, mev-boost). That matches no relay, so the sidecar rejects every request and the entry gets no bids, with nothing in the logs.

Four smaller issues came up along the way:

  • Credential leak in logs. The request-auth round logged builder URLs verbatim, and a URL can carry a basic-auth password or an API key in its path. That's the same class of leak goclient: keep beacon-node credentials out of errors and logs #3119 fixed for beacon addresses.
  • Contradicting comment. The BuilderConfig comment, the Builders env-description and the example config all said the whole config must be identical across operators. The URL isn't signed, though, and with a sidecar it naturally differs per operator. That only works if AuthData is set explicitly, because the default comes from the URL's hostname.
  • Stale pre-Gloas docs. The sections described the v1 blinded-block endpoints, a toggle that no longer exists, and a Prysm limitation of those v1 endpoints. SSV now fetches proposals with produceBlockV3 (/eth/v3/validator/blocks), which Prysm has served since its Deneb work in 2023. The "Edge cases outcomes" scenarios came from 2023 testing, before consensus switched to always running on blinded blocks (feat: blind all blocks (by iurii) #2569). In Scenario 3, operators without MEV rejected a builder block with "blinded blocks are not supported" and the round failed. That error no longer exists, and the value check doesn't depend on the builder setup.
  • Noisy pre-Gloas proposal failures. Every operator submits the decided block, but a beacon node can't unblind a block another operator's beacon node produced. Those expected misses logged an error, counted a failed submission and concluded the duty failed, even when the block landed.

What changed

  1. Sidecar warning. BuilderEntry.LocalHostWithoutAuthData reports when an entry omits AuthData and its URL host looks local: a loopback, private (IPv6 unique-local included), link-local or unspecified IP; a single-label name (localhost, a Docker service name); or a .localhost, .local or .internal name. resolveAndValidate warns at startup for each such entry, logging the entry index (builder_entry_index) and the hostname only. This is a warning, not an error, because a builder genuinely reached over a private network trips the same check.
  2. Masked builder URLs.
    • The masking that goclient.RedactBeaconAddr did (password, path and query values, as go-eth2-client masks Client.Address()) moves into format.RedactURL, so the beacon and builder paths share one implementation. RedactBeaconAddr behaves exactly as before; its existing tests are unchanged.
    • ResolvedBuilderEntry gains RedactedURL, computed once in ResolveBuilderConfig.
    • The request-auth round's three log sites use it. The builder-preferences submit still gets the URL verbatim.
    • Frozen request auths now point at the resolved entries, instead of copying four of their fields into a separate frozenBuilderRef.
  3. The identical-config rule. The URL is now its one exception, with the condition that operators whose hostnames differ set AuthData explicitly. The BuilderConfig comment, the Builders env-description (shown by ssvnode doc), config/config.example.yaml and the doc all say so.
  4. Pre-Gloas non-leader submits.
    • The TODO asking whether non-leaders should submit the decided block at all is replaced with a comment explaining why they do: ssv-spec's submitDecidedBlock has every operator submit, and it adds redundancy for builder blocks.
    • The runner now remembers the blinded SSZ of every block it fetches (before, only full ones) and compares the decided value with it.
    • A failed submit of a block another operator's beacon node produced logs at Info, skips the failed-submission counter, and concludes the duty with a new not_published outcome in ssv.runner.duty.outcome.
    • A failed submit of the operator's own block, full or blinded, still fails the duty as before. The Gloas path is unchanged.
    • The leader's cache-mismatch Debug log carries both block roots instead of two hex-encoded blocks, computed only when Debug is enabled, since hashing the full block would otherwise delay the submit.
  5. docs/EXTERNAL_BUILDERS.md:
    • A new "MEV-Boost and commit-boost after Gloas" section covers why a sidecar isn't needed and how to configure one anyway: one entry per relay, an explicit AuthData, a YAML example and a one-liner for the hex. It also says clusters can mix direct and sidecar setups, since only AuthData and the slot are signed. Because MEV-Boost's Gloas endpoints are still in review (Gloas flashbots/mev-boost#914), it notes that MEV-Boost's routing may still change.
    • A new bullet in the identical-config list says the URL isn't signed and may differ per operator.
    • The pre-Gloas sections now sit under "Before Gloas: MEV-Boost/commit-boost and relays", with a note that they stop applying at the fork. Their headings moved down one level.
    • In those sections, the blinded-block paragraph now names the endpoints SSV actually uses (produceBlockV3, /eth/v2/beacon/blinded_blocks, /eth/v2/beacon/blocks) and describes the current flow. Consensus always runs on the blinded form. The deciding round's leader submits its own full block, and every other operator submits the blinded block. The "if builder proposals are enabled" qualifiers are gone, because registrations are always scheduled and the beacon node decides blinded vs. full. The obsolete Prysm "Known issues" entry is removed.
    • The beacon-node builder links are fixed. Lighthouse's builders.html no longer exists, and the Prysm link only redirected to the generic parameters page. Both now point at the clients' MEV builder pages, and the list gains Grandine, Lodestar, Nimbus and Teku.
    • The three 2023 "Edge cases outcomes" scenarios are replaced by a short "Clusters with mixed builder setups" section. It covers three points:
      • Consensus works whatever each operator's builder setup.
      • The cluster decides on the round leader's block.
      • Some submissions are expected to fail when a beacon node can't get the payload: no builder, a sidecar on another relay, or another operator's local block. One successful submission is enough, and such misses report not_published.

Tests

  • TestBuilderEntry_LocalHostWithoutAuthData covers each class of local host, including an IPv4-mapped IPv6 loopback and a name with a trailing dot. It also checks that public hosts, an address just outside 172.16.0.0/12, an entry with explicit AuthData and an unparsable URL don't match.

  • Test_resolveAndValidate_builderLocalHostWithoutAuthData checks that only the local entry without AuthData warns, and that the log carries the hostname rather than the URL's credentials.

  • TestRedactURL covers the masking cases, including a username alone (a relay pubkey in MEV-Boost-style URLs), which is kept.

  • TestProposerPreferencesRunner_requestAuthLogsRedactedURL drives a request auth to quorum for a builder URL carrying a password and a path API key. It checks that no log line contains either one, and that the builder-preferences submit still receives the URL verbatim. I checked that it fails when the raw URL is logged again.

  • TestProposerRunnerProcessPostConsensusSubmitFailureOfAnotherOperatorsBlockIsNotPublished checks that a failed submit of another operator's block concludes not_published without returning an error. I checked that it fails without the classification. TestProposerRunnerProcessPostConsensusSubmitFailureOfOwnBlockFailsDuty checks that the operator's own block, full or blinded, still concludes failed. TestProposerRunnerProcessPreConsensusCachesOnlyBlindedBlockSSZ replaces the test asserting that blinded fetches weren't remembered. TestBlockRootField checks that a block and its blinded form log the same root, and that a root that can't be computed logs as an error.

  • go test for cli/..., protocol/v2/types/gloas, protocol/v2/ssv/... (spec tests included), beacon/goclient and utils/format passes, and so do golangci-lint on those packages and scripts/deadcode.sh.

  • E2E passed - [action log](put link here)

…r entries without AuthData

After Gloas the beacon node contacts each configured builder itself, so a
MEV-Boost or commit-boost sidecar is optional. An operator who keeps one
needs an entry per relay with URL set to the sidecar and AuthData set to
the relay's identity; omitting AuthData signs the sidecar's own hostname,
which matches no relay, so the entry silently gets no bids.

- Warn at startup for each builder entry that omits AuthData while its URL
  host looks local (loopback/private/link-local IPs, single-label names,
  .localhost/.local/.internal), logging the hostname only.
- Document running a sidecar after Gloas, that URLs may differ per
  operator, and scope the pre-Gloas sections to before the fork.
@iurii-ssv
iurii-ssv requested review from a team as code owners October 28, 2026 09:08
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.0%. Comparing base (0d043a1) to head (fc5d760).

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; no actionable issues were found.

Summary

Adds a startup warning for local-looking builder entries without explicit AuthData and explains sidecar setup after Gloas.

  • Startup warns when local builder hosts omit relay AuthData.
  • The builder guide explains when sidecars fit the Gloas flow.

Reviews (1) · Last reviewed commit: "gloas: document MEV-Boost/commit-boost a..." · Reviewed by Greptile

…refresh pre-Gloas docs

- Builder URLs may carry credentials (a basic-auth password, an API key
  in the path). The request-auth round logged them verbatim; it now logs
  them masked. The masking moves from goclient.RedactBeaconAddr into
  format.RedactURL, shared by both, and resolved builder entries carry a
  RedactedURL computed once at load.
- The BuilderConfig comment now says the URL is the one field that may
  differ per operator, matching the docs.
- The pre-Gloas docs described the v1 blinded-block endpoints and a
  Prysm limitation of them. SSV uses produceBlockV3 now, which Prysm
  serves, so the Prysm note goes and the endpoints are corrected; the
  "builder proposals enabled" toggle they mention no longer exists.
@iurii-ssv iurii-ssv changed the title gloas: document MEV-Boost/commit-boost after the fork, warn on sidecar entries without AuthData gloas: MEV-Boost/commit-boost docs, sidecar AuthData warning, masked builder URLs in logs Oct 28, 2026
…ork now

The "Edge cases outcomes" scenarios came from 2023 testing, before
consensus switched to always running on blinded blocks. Scenario 3 had
operators without MEV rejecting a builder block ("blinded blocks are not
supported") and failing the round; that check no longer exists, and the
value check doesn't depend on the builder setup. Scenarios 1 and 2
mostly recorded relay behavior rather than SSV's.

Replace them with a short section on clusters with mixed builder setups,
and correct the blinded-proposal paragraph: consensus always runs on the
blinded form, the deciding round's leader submits its own full block,
and every other operator submits the blinded block.
… explain non-leader submits

The URL exception to the identical-builder-config rule left out a trap:
AuthData defaults to the URL's hostname, so operators whose hostnames
differ (each pointing at its own sidecar) must set AuthData explicitly,
or the signed bytes diverge and the builder is silently disabled. Say so
in the BuilderConfig comment and the docs, and carry the qualified
exception into the Builders env-description and the example config,
which still required entries to be configured identically.

The Lighthouse builder link no longer resolves and the Prysm one
redirects to the generic parameters page; point both at their MEV
builder pages.

Replace the pre-Gloas TODO about non-leaders submitting the decided
block with why they do: ssv-spec has every operator submit, and it adds
redundancy for builder blocks. Their failures, when their beacon node
can't unblind the block, are expected.
… not_published

Before Gloas every operator submits the decided block, but a beacon node
can't unblind a block another operator's beacon node produced. Those
expected misses logged an error, counted a failed submission and
concluded the proposal duty failed, even when the block landed, which
made proposal-failure alerts noisy.

The runner now remembers the blinded SSZ of every block it fetches, not
only full ones, and compares the decided value with it. A failed submit
of another operator's block logs at debug, skips the failed-submission
counter and concludes the duty with a new not_published outcome. A
failed submit of the operator's own block, full or blinded, still fails
the duty.

Also list the builder docs of the other beacon clients (Grandine,
Lodestar, Nimbus, Teku) and describe the new outcome in the doc.
@iurii-ssv iurii-ssv changed the title gloas: MEV-Boost/commit-boost docs, sidecar AuthData warning, masked builder URLs in logs gloas: MEV-Boost/commit-boost docs, sidecar AuthData warning, masked builder URLs, not_published outcome Oct 28, 2026
- Log a failed submit of another operator's pre-Gloas block at Info, so
  the duty's Info log doesn't stop silently after "submitting block
  proposal"; the not_published outcome itself stays at Debug.
- Rename BuilderEntry.DefaultAuthDataOnLocalHost to
  LocalHostWithoutAuthData: it returns the bare host, not the default
  AuthData. The warning's index field becomes builder_entry_index, and
  its test matches the warning by message.
- Drop frozenBuilderRef: frozen request auths point at the resolved
  builder entries instead of copying four of their fields.
- Log the leader's cache-mismatch as two block roots instead of two
  hex-encoded blocks, and only when Debug is enabled, since hashing the
  full block would otherwise delay the submit.
- Docs: say the beacon node contacts the builders listed on the
  produceBlockV4 POST, and note that MEV-Boost's Gloas routing is still
  in review (flashbots/mev-boost#914).
- Shorten the comments this PR added or touched, and the example
  config's Builders note.

@y0sher y0sher left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One heads-up for after this lands: ssv.runner.duty.outcome gains a new not_published value, and pre-Gloas submit misses on another operator's block move out of failed. Any dashboard panel or alert that counts failed proposer duties will drop when this ships, so it's worth adding not_published wherever the total outcome breakdown matters.

@iurii-ssv
iurii-ssv merged commit 8777afc into stage Oct 28, 2026
10 checks passed
@iurii-ssv
iurii-ssv deleted the c/ssv-glamsterdam-mev-boost-affcd6 branch October 28, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants