Commit 050d6e9
fix(iceberg): Correct test setup to ensure delete files are created (#5864)
## Changes Made
The integration test
TestIcebergCountPushdown.test_count_pushdown_with_delete_files was
failing for the test_overlapping_deletes table because it incorrectly
enabled count pushdown.
The root cause was that the initial Spark write created multiple small
data files. Subsequent DELETE operations were optimized by Iceberg to
mark entire data files as removed instead of generating
position/equality delete files. As a result, Daft's _has_delete_files()
check did not find any delete files and incorrectly allowed the count
pushdown optimization.
This PR fixes the test by adding coalesce(1) to the Spark DataFrame
before writing the initial data for the test_overlapping_deletes table.
This ensures the data is written to a single Parquet file, forcing
subsequent DELETE operations to generate actual delete files. This
aligns the test's behavior with its intent, correctly disabling count
pushdown when delete files are present.
## Related Issues
#5863 5863
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
---------
Co-authored-by: root <root@bytedance>
Co-authored-by: huleilei <huleilei@bytedance>1 parent b200c59 commit 050d6e9
File tree
2 files changed
+22
-22
lines changed- daft/io/iceberg
- tests/integration/iceberg/docker-compose
2 files changed
+22
-22
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
321 | 321 | | |
322 | 322 | | |
323 | 323 | | |
324 | | - | |
| 324 | + | |
325 | 325 | | |
326 | 326 | | |
327 | 327 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
452 | 452 | | |
453 | 453 | | |
454 | 454 | | |
455 | | - | |
456 | | - | |
457 | | - | |
458 | | - | |
459 | | - | |
460 | | - | |
461 | | - | |
462 | | - | |
463 | | - | |
464 | | - | |
465 | | - | |
466 | | - | |
467 | | - | |
468 | | - | |
469 | | - | |
470 | | - | |
471 | | - | |
472 | | - | |
473 | | - | |
474 | | - | |
475 | | - | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
476 | 476 | | |
477 | 477 | | |
478 | 478 | | |
| |||
0 commit comments