Problem
A Table Diff test comparing two large tables across services can run the ingestion pod out of memory. Seen on Snowflake ↔ SQL Server, ~831k rows, ~105 compared columns, with nearly every row differing: the pod was OOM-killed at both 4 GiB and 8 GiB.
TableDiffValidator only needs counts from data-diff, but:
get_stats_dict() keeps every differing row in memory. It drains the diff into DiffResultWrapper.result_list, each row as strings for every compared column. HashDiffer also kept each segment's rows on its info tree. Memory grows with differing rows × columns: ~6.7 GiB at 400k rows × 105 columns in a local repro, which projects to ~13–14 GiB at 831k.- The threshold path never stops early.
calculate_diffs_with_limit iterates the wrapper, so it also keeps every row, and its if len(key_set) > limit: len(key_set) is a no-op, so it diffs the whole table even once the result is known.
- Every failing test diffs the tables a second time for a debug log sample.
logger.level is NOTSET (0) on this logger, so 10 if logger.level <= logging.DEBUG else 0 is always 10.
- Taking a sample never stops the diff. Closing the diff iterator after a sample (Collate's failed-rows sample, or the debug sample above) didn't stop data-diff's worker pool. The rest of the diff kept running in the background, piling results up in memory.
Fix
- collate-data-diff: https://github.com/open-metadata/collate-data-diff/pull/36 adds
get_stats_dict(retain_rows=False), stops the worker pool when the iterator is closed, and raises DataDiffDuplicateKeyError naming the table instead of a bare assert.
- OpenMetadata: use those, read the threshold path off the raw diff and stop it at the threshold, and sample for the debug log only when debug logging is enabled.
Problem
A Table Diff test comparing two large tables across services can run the ingestion pod out of memory. Seen on Snowflake ↔ SQL Server, ~831k rows, ~105 compared columns, with nearly every row differing: the pod was OOM-killed at both 4 GiB and 8 GiB.
TableDiffValidatoronly needs counts from data-diff, but:get_stats_dict()keeps every differing row in memory. It drains the diff intoDiffResultWrapper.result_list, each row as strings for every compared column. HashDiffer also kept each segment's rows on its info tree. Memory grows with differing rows × columns: ~6.7 GiB at 400k rows × 105 columns in a local repro, which projects to ~13–14 GiB at 831k.calculate_diffs_with_limititerates the wrapper, so it also keeps every row, and itsif len(key_set) > limit: len(key_set)is a no-op, so it diffs the whole table even once the result is known.logger.levelisNOTSET(0) on this logger, so10 if logger.level <= logging.DEBUG else 0is always 10.Fix
get_stats_dict(retain_rows=False), stops the worker pool when the iterator is closed, and raisesDataDiffDuplicateKeyErrornaming the table instead of a bareassert.