EVM Watcher Divergence Tests

node/pkg/watchers/evm/README.md

@@ -0,0 +1,137 @@
+# EVM watcher tests
+
+This package has two complementary groups of tests: unit and integration tests
+covering the watcher's behavior, and hash regression tests that pin the exact
+digests the watcher produces for checked-in transaction receipts.
+
+## Unit and integration tests
+
+| File | What it covers |
+| ---- | -------------- |
+| `watcher_test.go` | Message processing: `postMessage` dispatch, immediate publication, pending-message queueing, `processNewBlock` finality handling, `verifyAndPublish`, consistency-level handling, and block-time retry logic. |
+| `reobserve_test.go` | Reobservation request handling: invalid chain IDs, receipt errors, failed transaction status, skipped logs, deterministic ordering, large receipts, and transfer-verifier integration. |
+| `by_transaction_test.go` | Core bridge log validation (`isValidCoreBridgeMessagePublicationLog`): wrong contract, wrong topic, removed logs, malformed topics. |
+| `custom_consistency_level_test.go` | Custom consistency level (CCL) config parsing and effective-consistency-level handling. |
+| `chain_config_test.go` | Per-chain configuration: finality support, EVM chain IDs, mainnet contract addresses. |
+| `blocks_by_timestamp_test.go` | The block-by-timestamp cache used for CCQ. |
+| `ccq_test.go`, `ccq_backfill_test.go` | Cross-chain query (CCQ) request handling and backfill. |
+| `tron_integration_test.go` | Live integration against the Tron Nile testnet (requires network access). |
+| `watcher_test_helpers_test.go` | Shared mock connector and test helpers. |
+
+## Hash regression tests
+
+These fixtures protect the message hash used to match live observations with
+re-observations. The expected hashes are checked in and are constructed without
+calling `MessagePublication.CreateDigest()`, so an accidental change to watcher
+message construction is caught by the tests.
+
+### Source of truth
+
+`TestObservationReobservationParity` in
+`observation_reobservation_parity_test.go` loads the canonical receipt vectors
+and sends each receipt through both watcher paths:
+
+- live observation: `runMessageProcessor` / `postMessage`
+- re-observation: `runReobservationHandler` /
+  `handleReobservationRequest`
+
+For every receipt, the test requires both paths to emit exactly the same
+digests - compared without regard to order, but with every duplicate counted -
+and requires those digests to match the checked-in `expectedMessages[].hash`
+values. It also asserts that `CreateDigest()` and `VAAHash()` remain
+equivalent.
+
+The digest is the double Keccak-256 hash of the serialized VAA body fields:
+
+1. timestamp as a big-endian `uint32`
+2. nonce as a big-endian `uint32`
+3. Wormhole chain ID as a big-endian `uint16`
+4. 32-byte emitter address
+5. sequence as a big-endian `uint64`
+6. consistency level as a `uint8`
+7. payload bytes
+
+Receipt metadata such as transaction hash, block hash, block number,
+transaction index, log index, gas fields, bloom, and `IsReobservation` must not
+change the digest. `TestGeneratedReceiptGoldenVectorMetadataIndependence`
+checks this explicitly.
+
+### Receipt fixtures
+
+The tests load only these canonical files:
+
+- `testdata/generated_receipts.json`: 210 deterministic synthetic receipt
+  vectors containing 211 expected messages.
+- `testdata/real_receipts.json`: 200 Ethereum mainnet receipt vectors containing
+  202 expected messages. Two receipts contain two Wormhole Core events.
+
+Each vector has this shape:
+
+```json
+{
+  "name": "real-000-tx-2461b4bce349",
+  "comment": "real: Ethereum Wormhole Core ...",
+  "wormholeChainId": 2,
+  "blockTime": 1749806543,
+  "receipt": {},
+  "expectedMessages": [
+    {
+      "logIndex": 645,
+      "hash": "cfdc8c25256503257c9d66a22b43e00a1ce42445592e5e8653c684e219cae1b5"
+    }
+  ]
+}
+```
+
+`hash` is the observation-matching digest, encoded as 32 lowercase hexadecimal
+bytes without a `0x` prefix. `logIndex` makes message selection explicit for
+receipts containing unrelated logs or multiple Wormhole events.
+
+### Data provenance and hash construction
+
+The synthetic vectors were built from explicit event fields, ABI-encoded into
+`LogMessagePublished` receipt logs, and paired with independently calculated
+hashes. The real vectors were derived from full Ethereum receipts. Historical
+block timestamps were matched to the corresponding Wormholescan VAAs by
+transaction hash, emitter, and sequence because timestamp is part of the signed
+body.
+
+For both data sets, each expected hash was calculated by serializing the VAA
+body fields in protocol order and applying Keccak-256 twice. This calculation
+did not call `MessagePublication.CreateDigest()`, `VAAHash()`, or another watcher
+hash helper. All 200 previously recorded historical hashes matched this
+independent calculation. The full receipts also revealed two additional valid
+Wormhole Core events, which is why 200 real receipts contain 202 expected
+messages.
+
+The fixtures are therefore checked from two independent directions: the stored
+hashes come from direct protocol serialization, while the tests decode the
+receipt logs through the production ABI and construct `MessagePublication`
+objects through the watcher paths. A mistake in watcher field mapping,
+serialization, or event selection causes the produced digest to differ from the
+checked-in value.
+
+The generated corpus covers:
+
+- empty, binary, all-zero, 1-, 31-, 32-, 33-, and 4096-byte payloads
+- leading-zero emitter addresses
+- timestamps with non-round and high-bit values
+- chain ID `4004`
+- sequence boundaries including `0`, `uint64` max, and `2^53 + 1`
+- maximum `uint32` nonce
+- immediate, safe, finalized, custom, and historical consistency levels
+- unrelated logs, wrong-contract logs, and multiple Wormhole events in one
+  receipt
+
+`TestGeneratedReceiptGoldenVectorsCoverage` pins these properties so fixture
+changes cannot silently weaken the corpus.
+
+### Tests to run
+
+From `node`:
+
+```sh
+go test ./pkg/watchers/evm -run TestObservationReobservationParity -count=1
+go test ./pkg/watchers/evm -run 'TestGeneratedReceiptGolden|TestConsistencyLevelMatches' -count=1
+go test ./pkg/watchers/evm -count=1
+```

node/pkg/watchers/evm/by_transaction.go

@@ -3,8 +3,10 @@ package evm
 import (
 	"context"
 	"fmt"
+	"time"
 
 	"github.com/certusone/wormhole/node/pkg/watchers/evm/connectors"
+	"github.com/certusone/wormhole/node/pkg/watchers/evm/connectors/ethabi"
 
 	"github.com/certusone/wormhole/node/pkg/common"
 	eth_common "github.com/ethereum/go-ethereum/common"
@@ -24,7 +26,7 @@ var (
 // has the expected LogMessagePublished event topic, and has not been removed
 // due to a chain reorganization. This is called from both the real-time
 // subscription path, postMessage, and the reobservation path,
-// MessageEventsForTransaction, to ensure consistent validation.
+// MessageEventsForTransactionReobservation, to ensure consistent validation.
 func isValidCoreBridgeMessagePublicationLog(l types.Log, contract eth_common.Address) bool {
 	// SECURITY: Reject logs that have been flagged as removed due to a chain reorg.
 	if l.Removed {
@@ -41,33 +43,91 @@ func isValidCoreBridgeMessagePublicationLog(l types.Log, contract eth_common.Add
 	return true
 }
 
-// MessageEventsForTransaction returns the lockup events for a given transaction.
-// Returns the block number and a list of MessagePublication events.
+// validateTransactionReceipt checks that a transaction receipt was successfully retrieved and that
+// the transaction executed successfully. It returns a non-nil error describing the problem when the
+// receipt is unusable. This is the shared validation used by both the real-time subscription path
+// (postMessage) and the reobservation path (MessageEventsForTransactionReobservation).
+//
+// SECURITY: Bail early when the receipt status is anything other than 1 (success). In theory this
+// check isn't strictly necessary - a failed transaction cannot emit logs and will trigger neither
+// subscription messages nor have log messages in its receipt. However, relying on that invariant is
+// brittle - we connect to a lot of EVM-compatible chains which might accidentally break this API
+// contract and return logs for failed transactions. Check explicitly instead.
+func validateTransactionReceipt(receipt *types.Receipt, err error) error {
+	if receipt == nil || err != nil {
+		return fmt.Errorf("failed to get transaction receipt: %w", err)
+	}
+	if receipt.Status != gethTypes.ReceiptStatusSuccessful {
+		return fmt.Errorf("non-success transaction status: %d", receipt.Status)
+	}
+	return nil
+}
+
+// newMessagePublication builds a MessagePublication from a parsed LogMessagePublished event and
+// the validated timestamp of the block that included it. This is the single place where the EVM watcher
+// translates on-chain event data (the ethabi.AbiLogMessagePublished struct returned by the EVM
+// libraries) into a MessagePublication, so the real-time subscription path (postMessage) and the
+// reobservation path (MessageEventsForTransactionReobservation) stay consistent.
+//
+// isReobservation must be true when the message is being reconstructed from a reobservation
+// request and false for live observations; the caller states this explicitly so the field is set
+// at construction rather than mutated afterward. The emitter chain id is supplied by the caller
+// and is always the watcher's own hardcoded chain id (SECURITY: it must never be derived from
+// untrusted event data).
+func newMessagePublication(ev *ethabi.AbiLogMessagePublished, timestamp time.Time, chainId vaa.ChainID, isReobservation bool) *common.MessagePublication {
+	return &common.MessagePublication{
+		TxID:             ev.Raw.TxHash.Bytes(),
+		Timestamp:        timestamp,
+		Nonce:            ev.Nonce,
+		Sequence:         ev.Sequence,
+		EmitterChain:     chainId, // SECURITY: Hardcoded chain id from watcher
+		EmitterAddress:   PadAddress(ev.Sender),
+		Payload:          ev.Payload,
+		ConsistencyLevel: ev.ConsistencyLevel,
+		IsReobservation:  isReobservation,
+		Unreliable:       false,
+	}
+}
+
+// MessageEventsForTransaction returns the receipt, block number, and messages for a transaction.
+// The returned messages are not marked as reobservations.
 func MessageEventsForTransaction(
 	ctx context.Context,
 	ethConn connectors.Connector,
 	contract eth_common.Address,
 	chainId vaa.ChainID,
 	tx eth_common.Hash) (*types.Receipt, uint64, []*common.MessagePublication, error) {
+	return messageEventsForTransaction(ctx, ethConn, contract, chainId, tx, false)
+}
+
+// MessageEventsForTransactionReobservation returns the receipt, block number, and messages for a transaction.
+// The returned messages are marked as reobservations. The caller is responsible for finality checks.
+func MessageEventsForTransactionReobservation(
+	ctx context.Context,
+	ethConn connectors.Connector,
+	contract eth_common.Address,
+	chainId vaa.ChainID,
+	tx eth_common.Hash) (*types.Receipt, uint64, []*common.MessagePublication, error) {
+	return messageEventsForTransaction(ctx, ethConn, contract, chainId, tx, true)
+}
+
+func messageEventsForTransaction(
+	ctx context.Context,
+	ethConn connectors.Connector,
+	contract eth_common.Address,
+	chainId vaa.ChainID,
+	tx eth_common.Hash,
+	isReobservation bool) (*types.Receipt, uint64, []*common.MessagePublication, error) {
 
 	// Get transactions logs from transaction
 	// API only returns transactions that have been included in a block. Nothing in the mempool
 	receipt, err := ethConn.TransactionReceipt(ctx, tx)
-	if receipt == nil || err != nil {
-		return nil, 0, nil, fmt.Errorf("failed to get transaction receipt: %w", err)
-	}
-
-	// SECURITY
-	// Bail early when the transaction receipt status is anything other than
-	// 1 (success). In theory, this check isn't strictly necessary - a failed
-	// transaction cannot emit logs and will trigger neither subscription
-	// messages nor have log messages in its receipt.
-	//
-	// However, relying on that invariant is brittle - we connect to a lot of
-	// EVM-compatible chains which might accidentally break this API contract
-	// and return logs for failed transactions. Check explicitly instead.
-	if receipt.Status != gethTypes.ReceiptStatusSuccessful {
-		return nil, 0, nil, fmt.Errorf("non-success transaction status: %d", receipt.Status)
+	// SECURITY: Do not trust the logs of a transaction whose receipt could not be fetched or whose
+	// execution did not succeed. A failed transaction cannot emit logs, so a non-success receipt
+	// here means the RPC node is misbehaving; bail before we parse any events from it. See
+	// validateTransactionReceipt for the full rationale.
+	if valErr := validateTransactionReceipt(receipt, err); valErr != nil {
+		return nil, 0, nil, valErr
 	}
 
 	// Get block
@@ -98,20 +158,7 @@ func MessageEventsForTransaction(
 				tx.Hex(), ev.Raw.BlockNumber, blockTime, err)
 		}
 
-		message := &common.MessagePublication{
-			TxID:             ev.Raw.TxHash.Bytes(),
-			Timestamp:        timestamp,
-			Nonce:            ev.Nonce,
-			Sequence:         ev.Sequence,
-			EmitterChain:     chainId, // SECURITY: Hardcoded to watcher chain id
-			EmitterAddress:   PadAddress(ev.Sender),
-			Payload:          ev.Payload,
-			ConsistencyLevel: ev.ConsistencyLevel,
-			IsReobservation:  false,
-			Unreliable:       false,
-		}
-
-		msgs = append(msgs, message)
+		msgs = append(msgs, newMessagePublication(ev, timestamp, chainId, isReobservation))
 	}
 
 	return receipt, receipt.BlockNumber.Uint64(), msgs, nil