Skip to content

fix: retry the Go module fetch in every Go sample's Dockerfile - #245

Merged
slayerjain merged 4 commits into
mainfrom
fix/retry-go-module-fetch-in-dockerfiles
Oct 17, 2026
Merged

slayerjain merged 4 commits into
mainfrom
fix/retry-go-module-fetch-in-dockerfiles

Conversation

@slayerjain

@slayerjain slayerjain commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Problem

proxy.golang.org now and then resets an HTTP/2 stream partway through a module download:

go: golang.org/x/text@v0.31.0: read "https://proxy.golang.org/...": stream error: stream ID 187; INTERNAL_ERROR; received from peer

The next attempt succeeds. But every Go sample's Dockerfile here fetched its modules exactly once, using a bare go mod download, a bare go mod tidy, or a go build that fetches what it needs. One reset therefore fails the image build. It also fails the CI lane of whichever keploy repo was running docker build on the sample: keploy/keploy's workflows and other keploy CI clone this repo at main and build these Dockerfiles. This repo's own build.yml has the same exposure through its bare go build ./....

Changes

  1. fix: retry the Go module fetch in every Go sample's Dockerfile (27 Dockerfiles). Each fetch now runs in a loop: 5 attempts, 5-20s apart, then the build fails.

    RUN n=0; until go mod download; do n=$((n+1)); [ "$n" -lt 5 ] || exit 1; echo "go mod download failed (attempt $n), retrying in $((n*5))s"; sleep $((n*5)); done
    
    • The 19 bare go mod downloads and go-grpc's two go mod tidys run in the loop.
    • echo-sql, gin-redis, mux-mysql, sse-svelte and users-profile fetched inside go build. They now run the looped go mod download first.
    • nethttp-mysql's go mod download || true ignored a failed fetch and left it to an un-retried go mod tidy. Both are looped now.

    A module fetch is idempotent and fails only because of the network, so retrying it is safe. A missing module or a bad go.mod fails every attempt and still fails the build.

  2. fix(gin-redis, gin-pulsar): build them on a Go their go.mod accepts. Neither image built before this PR. gin-redis's go.mod needs Go 1.23 but the image was golang:1.20; gin-pulsar's needs Go 1.25 but the image was golang:1.23.

  3. ci: retry each sample's module fetch before building it. build.yml now runs go list -deps ./... in a loop, the same 5 attempts, before each go build ./.... go list -deps loads the same packages go build loads, so it fetches nothing more.

  4. ci: check that every Go module fetch in a Dockerfile runs in a bounded retry loop. .github/scripts/check_go_fetch_retried.py, run by a new dockerfile-fetch job in build.yml after a self-test on 26 fixtures, reads every Dockerfile. In each stage:

    • every fetch (go mod download/tidy/vendor, go install/get/run at a version) must be the condition of an until loop that fails the build when it runs out;
    • the first go build/test/run/vet/generate/list must come after a looped module fetch, in the stage, in one it is built FROM, or in one whose module cache it copies;
    • a RUN in exec form or with a here-document that runs go is refused, since it cannot be verified.

    It flags all 27 Go Dockerfiles on main (50 lines) and none on this branch. keploy enterprise pipeline 11675 failed on go-grpc's RUN go mod tidy ("stream error: stream ID 167; INTERNAL_ERROR"), which this PR loops.

Rebased onto main (07c5ba9) on 2026-10-17. No Go Dockerfile added since this PR was opened is missed: the guard reads them all.

Verification

  • All 27 changed Dockerfiles were built on a test machine: 25 built. gin-pulsar and gin-redis failed for the same reason the unchanged Dockerfiles fail, and both build after commit 2. docker build --check reports no warnings on any of them.
  • A GOPROXY was set up that cuts the first download of one module partway:
    • The unchanged go-docker-timefreeze and go-grpc server Dockerfiles fail with unexpected EOF (on golang-jwt/jwt v5.3.0 and on grpc v1.69.2). The looped ones recover on the second attempt.
    • The same holds for build.yml's step: the old step fails on go-docker-timefreeze, and the new one recovers.
  • build.yml's step was run as written, with Go 1.27.0 and an empty module cache: all 44 modules build, both with and without this change.
  • keploy/keploy's and keploy's other CI were searched for sed edits to these Dockerfiles. Two edit them: one rewrites go-docker-timefreeze's RUN CGO_ENABLED=0 GOOS=linux go build -o /main . line, and keploy's gin_mongo macOS script rewrites gin-mongo's EXPOSE 8080. This PR leaves both lines unchanged.

proxy.golang.org now and then resets an HTTP/2 stream partway through a
download:

  go: golang.org/x/text@v0.31.0: read "https://proxy.golang.org/...":
      stream error: stream ID 187; INTERNAL_ERROR; received from peer

The next attempt succeeds. But each Go sample's Dockerfile fetched its
modules exactly once, with a bare `go mod download`, a bare `go mod
tidy`, or a `go build` that fetches what it needs. So one reset failed
the image build, and with it the CI lane of whichever repo was building
the sample: the CI of keploy/keploy and of other keploy repositories
clones this repo and runs `docker build` on these Dockerfiles.

Each fetch now runs in a loop: 5 attempts, 5-20s apart, and then the
build fails. A module fetch is idempotent and fails because of the
network, so retrying it is safe. A missing module or a bad go.mod fails
every attempt and still fails the build, about 50s later.

- The 19 bare `go mod download`s and go-grpc's two `go mod tidy`s run
  in the loop.
- echo-sql, gin-redis, mux-mysql, sse-svelte and users-profile fetched
  inside `go build`. They now run the looped `go mod download` first.
- nethttp-mysql's `go mod download || true` ignored a failed fetch and
  left it to an un-retried `go mod tidy`. Both run in the loop now, so
  a fetch that fails five times fails the build.

Built all 27 Dockerfiles: 25 build, and gin-pulsar and gin-redis fail
for the same reason as before this change: their image's Go is older
than their go.mod requires, which a separate commit fixes. With a GOPROXY that
cuts the first download of one module partway, the original
go-docker-timefreeze and go-grpc server builds fail ("unexpected EOF"
on golang-jwt/jwt v5.3.0 and on grpc v1.69.2), and the looped ones
recover on the second attempt.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Neither image built. gin-redis's go.mod says `go 1.23.0` and `toolchain
go1.23.1`, which the Go in golang:1.20 cannot parse:

  go: errors parsing go.mod:
  /app/go.mod:3: invalid go version '1.23.0': must match format 1.23
  /app/go.mod:5: unknown directive: toolchain

gin-pulsar's go.mod says `go 1.25.0`, which golang:1.23 refuses:

  go: go.mod requires go >= 1.25.0 (running go 1.23.12; GOTOOLCHAIN=local)

Each now builds on the Go minor release its go.mod requires,
golang:1.23 and golang:1.25, and both images build. gin-pulsar's README
asked for Go 1.23+ as well; it now asks for 1.25+.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
build.yml runs `go build ./...` in every top-level sample with a
go.mod, and that build fetched the sample's modules from
proxy.golang.org with no retry. One reset stream ("stream error: stream
ID N; INTERNAL_ERROR; received from peer") would fail the whole job.

Each sample now runs `go list -deps ./...` first, in a loop: 5
attempts, 5-20s apart, and then the job fails. `go list -deps` loads
the packages `go build` loads, so it fetches the same modules and no
more. `go mod download` would also fetch every module go.mod requires,
whether the build needs it or not.

Run as build.yml runs it, with Go 1.27.0 and an empty module cache, all
44 modules build, both with and without this change. With a GOPROXY
that cuts the first download of golang-jwt/jwt partway, the old step
fails on go-docker-timefreeze ("unexpected EOF"), and the new one
recovers on the second attempt.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain force-pushed the fix/retry-go-module-fetch-in-dockerfiles branch 5 times, most recently from 68393bc to 3752fa5 Compare October 17, 2026 12:03
…d retry loop

keploy's CI clones this repo at main and builds these Dockerfiles, so a
Go module fetch made only once in one of them fails a keploy lane on a
reset download: keploy enterprise pipeline 11675 failed on go-grpc's
`RUN go mod tidy` ("stream error: stream ID 167; INTERNAL_ERROR").

.github/scripts/check_go_fetch_retried.py reads every Dockerfile. It
reads continuations as one instruction, with each problem at its first
line, and drops comment and blank lines inside a continuation, as Docker
does. Quoted text, arithmetic and comments are blanked first. go is
found alone or by its path (/usr/local/go/bin/go), in backticks, and in
the script of an `sh -c`/`bash -c` (any flags: -e, -o pipefail,
--login, -c --). A here-document's lines are skipped by line number, so
a continuation inside one cannot run its delimiter into the next line.

In each stage:
- every fetch must be the condition of an `until` (or `while !`) loop
  (under env assignments and a `timeout N`, if any) in the RUN's own
  shell, not a subshell, an sh -c, a pipeline or the background (nor in
  a brace group that is), whose exit leaves only that shell. In the loop's own body (to its `; done`),
  a counter `n=$((n+1))`, never reset there, and a test of it that fails
  the build after 2 to 10 tries: `[ "$n" -lt N ] || exit K` (also
  `test`, `|| { ...; exit K; }`, `-ge N ] && exit K`, `if ...; then
  ...; exit K` within the then-block), K > 0, the test and the count
  outside any subshell. A fetch is `go mod download`, `tidy`,
  `vendor`, any `go get` (it adds and upgrades requirements), or `go
  install`/`run` of a module at a version. One with -e is refused: it
  succeeds on a fetch that failed, so its loop never retries.
- the first build must come after a loop of `go mod download`/`tidy`/
  `vendor` for its module: in its WORKDIR or one above it, in the stage,
  in one it is built FROM (with that stage's WORKDIR), or in one whose
  whole module cache it COPYs to the same place, and no ENV or ARG has
  moved GOMODCACHE or GOPATH from the image's own since. A build is go build, test, run,
  vet, generate, list, tool, mod why, mod graph, or install of the
  module itself: it fetches what the module cache lacks, once. A tool installed at a
  version fetches nothing of the module's.
- a loop in a RUN with a cache mount at the module cache fills that
  mount. Only a RUN with the same mount (by id, else target) sees it,
  and such a mount hides the image's own module cache.
- a RUN this cannot read is refused: exec form, a here-document (its
  delimiter read as written, quoted or not), or go in a quoted string
  run by anything but echo or printf (su -c, eval), or piped into a
  shell (or through xargs into one). A COPY or ADD here-document's body
  is skipped too. Not read at all: what a script or Makefile the RUN
  runs does.

The new dockerfile-fetch job in build.yml runs it, after a self-test on
120 fixtures. 71 of 72 mutants of the rule fail it; the other
changes nothing a real Dockerfile shows (a COPY of three paths, the
first two the same).

It flags all 27 Go Dockerfiles on main (50 lines) and none with this
branch's loops.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain force-pushed the fix/retry-go-module-fetch-in-dockerfiles branch from 3752fa5 to 903624c Compare October 17, 2026 12:10
@slayerjain
slayerjain merged commit 9593f89 into main Oct 17, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant