Overwrite keeps deleted-only manifests, so the manifest list grows by one on every overwrite #4165

Apache Iceberg version

0.11.1. main has the same logic.

Please describe the bug 🐞

When an overwrite removes files explicitly (update_snapshot().overwrite() with delete_data_file), the new snapshot keeps every earlier manifest that holds only DELETED entries. Each overwrite also writes a manifest recording its own deletions, so the manifest list grows by one per commit and only one manifest ever holds live files. Each later commit and scan plan opens all of them.

In _OverwriteFiles._existing_manifests, each parent manifest is read with fetch_manifest_entry(discard_deleted=True). A manifest whose entries are all DELETED comes back empty, so found_deleted_entries is empty and the manifest takes the "intercept is empty" branch, where it is kept unchanged.

Java drops these. MergingSnapshotProducer.apply keeps a manifest only if hasAddedFiles() || hasExistingFiles() || snapshotId() == snapshotId(). The equivalent check in pyiceberg would skip parent manifests with no added or existing files before reading them.

Table.overwrite() is not affected. It deletes through _DeleteFiles before appending.

Repro (local SQL catalog):

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.io.pyarrow import _dataframe_to_data_files

wh = tempfile.mkdtemp()
catalog = SqlCatalog("repro", uri=f"sqlite:///{wh}/catalog.db", warehouse=f"file://{wh}")
catalog.create_namespace("ns")
table = catalog.create_table("ns.t", schema=pa.schema([("id", pa.int64())]))
table.append(pa.table({"id": [0]}))
for i in range(1, 11):
    table = catalog.load_table("ns.t")
    with table.transaction() as tx:
        new_files = list(_dataframe_to_data_files(tx.table_metadata, pa.table({"id": [i]}), table.io))
        with tx.update_snapshot().overwrite() as producer:
            for task in table.scan().plan_files():
                producer.delete_data_file(task.file)
            for f in new_files:
                producer.append_data_file(f)
    table = catalog.load_table("ns.t")
    manifests = table.current_snapshot().manifests(table.io)
    live = sum(1 for m in manifests if m.added_files_count or m.existing_files_count)
    print(f"overwrite {i}: {len(manifests)} manifests, {live} with live files")

Output: overwrite 1: 2 manifests, 1 with live files up to overwrite 10: 11 manifests, 1 with live files.

In production, a table overwritten a few hundred times reached 388 manifests, 387 of them deleted-only. Each overwrite commit went from about 3 s to about 24 s on S3.

Willingness to contribute

• I can contribute a fix for this bug independently

🧑‍💻🤖 — posted via Claude Code