[yawkat]

LZ4BlockInputStream sized its output buffer from the originalLen and its compressed buffer from the compressedLen of a block header before reading any of the block data. A 21 byte header could therefore make it allocate about 64 MiB, which stays referenced while the underlying stream stalls (e.g. a peer that sends a header and then nothing). LZ4FrameInputStream did the same with two buffers of the frame's maximum block size (up to 8 MiB) for a block header of any size. The allocations are bounded by the maximum block size of the formats, but not by the amount of data actually received.

Changes

• LZ4Utils.readFullyGrowing reads block data into a buffer that only grows as the data arrives, to at most twice the data read so far (at least min(len, 64 KiB)). Beyond the requested length, it grows geometrically up to the maximum block size, so that a sequence of slightly larger blocks (e.g. from flushing growing messages) does not reallocate every time.
• The output buffer of a compressed block is only allocated after the compressed data was read, sized for the maximum LZ4 compression ratio of 255 (MAX_COMPRESSION_RATIO, moved from LZ4DecompressorWithLength to LZ4Constants). LZ4BlockInputStream now rejects block headers whose original length exceeds that ratio; valid LZ4 data cannot exceed it.
• The output buffer growth of LZ4BlockInputStream is capped at the block's maximum size (it could previously reach 48 MiB).
• A truncated block in LZ4FrameInputStream is still reported as a plain IOException("Stream ended prematurely"), not as an EOFException that callers may take for a clean end of data.
• An Error such as OutOfMemoryError while reading a block now also makes both streams fail permanently, since the block was already partially consumed.
• The acceptOversizedBlocks javadoc no longer warns about attacker-controlled allocations: oversized blocks can now only allocate what was actually read.

With this, the memory a stream holds is proportional to the data it received (at most 255× for highly compressible blocks, which is inherent to decompression). Measured with a header-only input: 67 MB → 64 KiB for the block stream, 8 MiB → 64 KiB for the frame stream.

Tests

• Truncated blocks claiming the maximum block size allocate less than 1 MiB (both streams, compressed and uncompressed).
• Header rejection at exactly 255:1 / just above.
• Maximally compressible data (all zeros) at the maximum block size round-trips with the fast and HC compressors, to make sure the ratio check does not reject valid data.
• Sequences of slowly growing blocks do not reallocate per block.
• testBlockBuffersReusedAcrossFrames now uses blocks that need the full 4 MiB output buffer; with the smaller buffers it could no longer detect missing reuse.

🤖 Generated with Claude Code