[iandees]

The weekly fabric jobs in openaddresses/batch have been running out of memory in tile-join when joining their geographic shards (parcels was OOM-killed at 58GB within about 6 minutes of starting the join, addresses ran for hours without writing a tile and then died at 110GB). Raising the memory limits and capping threads did not help, and the version of tippecanoe in this image turned out to be the cause.

I reproduced it locally in an Alpine/musl container (same libc as this image) with real parcels data: 37M features from 138 sources, tiled into 16 shards and joined with the same flags the fabric job uses (--no-tile-size-limit, 4 threads):

tile-join	result
2.31.0 (current)	anonymous memory reached about 14-16GB within 60 seconds and stayed there; after 40 minutes the output file was 28KB, so I killed it
2.79.0	completed in 39 minutes, peak anonymous memory 2.4GB, 7.09GB output

That input is about 10% of the production parcels data, and 2.31.0 allocated about 2.4x the tile data up front, so the full-size join would want well over 58GB, which is consistent with the OOM kills.

2.79.0 is the newest tagged release; it also contains several tiling memory reductions between 2.37 and 2.75.

This image also runs tippecanoe for the per-source slippy map in openaddr/slippymap.py (tippecanoe -l dots -r 3 ...), so I compared 2.31.0 and 2.79.0 on that exact command line with 300k points: identical tile counts at every zoom, 486,991 vs 486,985 features across all tiles, file sizes within 1%, and 2.79.0 was faster (20s vs 28s) though it used more memory (147MB vs 28MB). The existing slippymap tests mock subprocess.Popen, so they don't cover this.

Checked:

• the image builds on the pinned GDAL base and reports tippecanoe v2.79.0
• python3 test.py in the built image: 126 tests, OK

Not checked:

• the full-size fabric join. Needs a release of this image and a real run.
• other flags used outside this repo (the batch repo calls tippecanoe and tile-join from task/fabric.js with its own options)

After merging this needs a version bump and tag so the ECR workflow publishes the image, and then task/Dockerfile in openaddresses/batch needs to point at the new version.