Design: Bind Web cluster hostname aliases to existing advertised-endpoint collectors

[barisdincer]

Related issue: #544 — Cluster node hostname aliases do not share advertised-IP metrics collectors
Suggested category: Design
Status: Proposal for maintainer review; implementation has not started.

Problem

In Web mode, a cluster connection can use a hostname while cluster discovery advertises the same node by IP address. Connection identity and metrics collector identity then differ:

Identity	Generic example
User connection	node.example.com:6379, DB 0
Connection ID	node-example-com-6379-db0
Advertised endpoint	192.0.2.10:6379
Endpoint-derived collector ID	192-0-2-10-6379

Removing the DB suffix from the connection ID does not establish which advertised node the hostname represents.

The paths documented in #544 expose two consequences:

• Connecting through the hostname first can start a hostname-keyed collector before discovery adds the advertised-IP collector.
• Connecting through the IP first and then the hostname can reuse a cached cluster client without establishing an alias-to-collector binding.

Collector-backed Dashboard, Monitor, Activity and configuration paths consequently cannot consistently locate the existing collector.

This proposal addresses hostname/IP alias binding to metrics collectors. The separately reported Metrics server responded with 500 has no confirmed root cause and is not claimed to be resolved by this design.

The proposal follows ArgusLi’s feedback on closed PR #545: a binding must follow revalidation and topology changes, and cached client reuse must not transfer another connection’s metrics identity.

The source baseline reviewed is main at 64f5fb4.

Invariants

1. Connection, session and DB identities remain unchanged.
2. A cached/shared GlideClusterClient is not evidence of a hostname alias’s node identity.
3. An alias binding is never copied from another connection.
4. Web cluster connections do not start a hostname collector before cluster detection and discovery.
5. Collector ID, spawn host, lookup and close all use the same advertised endpoint.
6. Bindings do not cache collector URIs; dispatch and retry consult the current collector registration.
7. Connect, retry, reset and background refresh revalidate alias bindings.
8. Revalidation can observe DNS changes even when topology is unchanged.
9. Stale asynchronous probe results cannot overwrite newer binding state.
10. Ambiguous or unverifiable mappings fail closed for collector-backed operations.
11. There is no hostname collector fallback.
12. Native IDs can be matched against both primary and nested replica advertised endpoints.
13. Closing one alias does not close a collector still required by another active alias.

Proposed design

The binding model is:

hostname alias → independent node-local probe → native node ID → current topology advertised endpoint → existing endpoint-keyed collector

Identity sources

Each identity retains its existing purpose:

Identity	Source of truth
Connection/session/DB	Existing connection and authorization state
Alias’s observed native node	Independent node-local probe
Native node’s advertised endpoint	Current topology for the connection’s cluster context
Collector registration and URI	Existing endpoint-keyed collector registry

The native node ID is binding evidence and topology metadata. It does not become a new collector registration key.

Independent alias verification

An alias uses a fresh node-local probe with its connection settings to obtain the identity of the node that actually answers. CLUSTER MYID supplies that node’s native ID.

This probe does not use a shared cluster client as identity evidence. A separate DNS lookup performed after connection establishment is also not treated as proof of where that connection went.

The observed native ID must map unambiguously to a usable advertised endpoint in the current topology. Discovery retains primary native IDs as well as replica native IDs. Repeated appearances of the same node across slot ranges are deduplicated; contradictory associations are rejected.

The initial supported mapping contract is the #544 case: a hostname seed maps to a cluster node advertising a usable IP address and port. Unsupported or indeterminate advertised addresses do not trigger a hostname collector fallback.

Shared collector lookup

Collector-backed consumers use a shared backend resolution path:

1. Validate the connection’s current binding.
2. Obtain its advertised-endpoint collector key.
3. Read the current collector registration.
4. Dispatch using that registration’s current URI.

Connection-scoped response IDs, authorization and DB context remain unchanged.

Monitor state updates reach authorized hostname/IP alias watchers bound to the same collector. Recipient WebSockets are deduplicated without changing their outward connection identity.

Lifecycle

Connect and cached client reuse

Cluster detection and discovery precede collector startup for a Web cluster seed. The independent alias probe establishes the binding, and collector startup uses the resulting advertised endpoint.

Existing cluster transport reuse can remain. Both an exact cache hit and reuse through another connection still require the alias’s own verification; neither inherits an existing alias binding.

An explicitly connected replica can be matched through its native ID. This does not require enabling collectors for every discovered replica.

Refresh, DNS changes and failover

Alias revalidation runs through the existing backend refresh lifecycle, including refreshes where topology compares equal.

A healthy, long-lived probe alone would not observe a changed DNS target. Revalidation therefore establishes a fresh node-local connection using the original alias and connection settings. It publishes a binding only after checking the result against current topology.

Topology refresh updates native-ID-to-endpoint associations, including role changes and replica promotion. Bindings to nodes that disappear or become ambiguous are invalidated.

This provides periodically revalidated node semantics, not an instantaneous DNS-following guarantee.

Retry, reset and asynchronous work

Connect, retry and reset must not rely on an unvalidated inherited binding. Retry revalidates the alias and reads the current collector registration.

An operation retains its intended binding. If revalidation changes the alias to another native node or endpoint, the existing operation fails rather than silently moving to that node. A subsequent operation can use the newly verified binding.

Backend-local revision or equivalent existing-state checks prevent probes and callbacks started before reset, close or a newer refresh from publishing stale results. This is local lifecycle bookkeeping, not a collector wire generation protocol.

When revalidation is required, an unverifiable binding cannot continue dispatching through its previous target.

Collector spawn, registration, ping and close

The existing collector wire protocol remains unchanged:

• Startup uses the advertised endpoint for both collector key and child host/port.
• Concurrent aliases share the existing endpoint startup/registration path.
• Registration and ping retain current authentication and readiness behavior.
• Bindings store no collector URI; a replacement registration is read at dispatch.
• Missing or unregistered collectors produce controlled unavailable/not-ready results.
• Cleanup accounts for active canonical bindings and existing Web lifecycle ownership.
• Delayed cleanup for an old collector must not remove its replacement.

Closing one alias releases its binding. It does not terminate a collector still required by another alias or by existing collector ownership policy.

Failure behavior

“Fail closed” here means refusing collector-backed dispatch when the binding cannot be verified. It does not redefine successful cluster connectivity as proof of metrics identity.

Condition	Behavior
DNS or probe failure	Binding unavailable; no hostname fallback
CLUSTER MYID unavailable or unauthorized	Controlled unavailable result
Native ID absent from current cluster topology	Reject binding
Conflicting native-ID/endpoint associations	Reject binding
Unsupported advertised address	Reject this binding path
Known ambiguity involving multiple possible native nodes	Do not arbitrarily select a collector
Collector spawning, unavailable or not registered	Existing not-ready/failure behavior
Alias changes target during an operation	Do not retarget that operation
Probe completes after reset, close or a newer validation	Discard result
Collector respawns	Resolve current registration; do not reuse a cached URI

A successful probe establishes which node answered that probe. It does not prove that every DNS answer for a shared or load-balanced hostname denotes the same node. Such ambiguity must not be hidden by selecting the first DNS address.

Scope and non-goals

The primary scope is Web deployment behavior for #544.

Expected implementation boundaries are:

• Backend connection/discovery and refresh state.
• Metrics orchestrator startup, lookup and ownership cleanup.
• Collector-backed action consumers and watcher fan-out.

Consumer changes should route through shared resolution rather than implement separate DNS or identity logic.

Electron and Kubernetes paths require regression analysis and validation because they share backend code. Their existing deployment-specific collector behavior must remain intact. In particular, this proposal does not redesign Kubernetes sidecar identity or require manifest changes.

Other non-goals:

• Changing connection, session, DB or frontend identity formats.
• Migrating collector keys or stored metrics to native node IDs.
• Adding collector native-ID metadata, wire generations or heartbeat capabilities.
• Adding operation-socket identity guards.
• Redesigning general command routing or direct cluster-routed configuration commands.
• Weakening authentication, authorization or TLS verification.
• Diagnosing or claiming to fix the separately reported HTTP 500.

Alternatives considered

Alternative	Assessment
Selected: independent alias probe → topology endpoint → existing collector	Addresses alias verification and lifecycle without changing collector protocol or key format. Limited to the existing endpoint transport guarantee.
B: collector native identity/generation plus operation-boundary validation*	Second choice if maintainers require native identity verification on the actual operation connection. Broader protocol, reconnect and deployment changes.
Backend run_id matching and revalidation	A process observation does not establish the identity of a later operation connection. Does not close reconnect/TOCTOU risk.
Canonical native-node collector identity redesign	Requires wider key/storage/lifecycle migration. Native keys alone still do not verify the actual operation socket. Excessive scope for #544.

B* would require validating identity on the connection that executes the operation and handling reconnect/replay correctly. An identity check followed by an operation on a different connection is insufficient. Its Monitor socket feasibility remains an investigation item.

Known limitations and trade-offs

This design does not guarantee operation-time native identity when the same IP:port is reassigned to another native node. A verified binding can become stale before dispatch or reconnect; Monitor can open another connection to that endpoint.

This risk also exists in the current direct-IP collector model. The proposal does not strengthen that transport guarantee and must not claim that it does.

Additional limits:

• DNS changes become observable through refresh/retry, not immediately or according to DNS TTL.
• Revalidation adds bounded node-local connection and command overhead and can briefly make collector-backed operations unavailable.
• Probe permissions and TLS compatibility are required. An advertised IP that cannot satisfy existing TLS verification produces a controlled failure.
• Invalidating a binding cannot retract an already dispatched request or guarantee exactly-once effects.
• Missing replica advertisements or unverifiable topology yield unavailability rather than guessed mappings.

Testing plan

The following tests are planned after design and implementation approval. They have not been implemented or run for this proposal.

Area	Planned regression coverage
Alias creation	IP-first, hostname-first and concurrent aliases share one advertised-endpoint collector
Existing IP behavior	Direct-IP connections preserve existing lookup and lifecycle behavior
Cache reuse	Exact cache hits and cross-alias shared clients still require independent verification
DNS lifecycle	Target changes with unchanged topology; failure removes dispatch eligibility; no hostname fallback
Topology and roles	Primary/replica matching, failover, endpoint changes, removed nodes and repeated slot entries
Ambiguity	Conflicting mappings, unsupported endpoints and different-port cases do not select unrelated collectors
Async lifecycle	Out-of-order probes, reset, close and newer refresh cannot accept stale results
Collector replacement	Retry uses current registration; delayed cleanup cannot delete a replacement
Ownership	Closing one alias preserves another active alias’s collector
Consumers	Dashboard, Activity, Monitor and collector-backed config share resolution; watcher delivery is authorized and deduplicated
Compatibility	Session/DB/reply identities, standalone behavior, Electron, Kubernetes and auth/TLS/IAM paths remain intact

A disposable Web integration environment should exercise a real cluster and controlled hostname changes, including unchanged-topology DNS changes and collector restart. Examples and fixtures will use generic addresses and names.

After implementation, repository-required unit, integration and lint checks will be run. Results will distinguish mocked logic, live integration and deployment-mode evidence. No test result will be presented as resolving the unconfirmed HTTP 500.

Questions for maintainers

1. Is the Web-focused scope acceptable with the explicit existing direct-IP limitation, or is operation-time native identity verification required? If required, B* needs a broader design.
2. Is periodic revalidation through the existing refresh lifecycle sufficient, or should specific actions require additional immediate validation?
3. For publishing a binding, is an independently observed native ID plus current topology association sufficient, or should the advertised endpoint also receive a node-local identity check? The latter would still leave the documented operation-time race.
4. Should unavailable CLUSTER MYID permission disable collector-backed alias operations while preserving normal cluster connectivity?
5. Is retaining existing collector coverage while supporting explicitly connected replica aliases acceptable?

Requested decision: Approval of this bounded design, or clarification of the guarantees required before implementation begins.

@ArgusLi, could you review this design for #544, particularly the lifecycle guarantees and the explicitly documented operation-time identity limitation?

[ArgusLi]

Hi @barisdincer, I really appreciate you writing up this design! It's well written and makes sense. However, I'm going to be taking on this piece of work as part of a larger refactor #552. I will be referencing this design as I write up my own design, so could you do me the favor of reviewing my design once it's out? I'll tag you.

[barisdincer]

Thanks @ArgusLi. Absolutely, I'd be happy to review it. Looking forward to seeing how you incorporate the approach into the broader refactor.