Data quality & master data
Safe Bulk Import
Loaded 2.6 million product records into a live database under three rules: only fill empty fields, never delete, and never write anything ambiguous.
Data engineer
Jul–Aug 2026
In numbers
- product records and 56,744 customer links
- 2,627,085
- product records and 56,744 customer links
- records deleted
- 0
- records deleted
- ambiguous matches held for review
- ~23K
- ambiguous matches held for review
- suppliers audited for completeness
- 961,514
- suppliers audited for completeness
The problem
Products, certifications and customer relationships extracted from the web archive had to go into the live supplier database. Data from 496,000 domains needed matching to existing suppliers first. One mistake in live data would have broken the profiles buyers were looking at.
What I did
- 1
Resolve once, reuse everywhere
I resolved which supplier each of the 496,000 domains belonged to in a single mapping table that every later step used, so the matching logic lived in exactly one place. Before writing anything, a dry run reported exactly what would go where.
- 2
Three rules
Never overwrite: only fill empty fields (175,893 were filled). Never delete: retire stale records by marking them inactive so they can come back (7,484 were). Never guess: if a domain matched several companies, an inactive one, or none at all, it wasn't written; it went to review. Certifications matching our catalogue went in directly (9,422); unknown ones were flagged for review (6,423).
- 3
Same rules for clean-up
I measured field completeness across 961,514 suppliers and ran a spam clean-up the same way: 13,000 candidates found with SQL, confirmed with Bedrock, and 1,056 deactivated without deleting a single row.
Result
2.6 million product records and 57,000 customer relationships went live without deleting a single row. The 23,000 ambiguous matches were never written; they sit in a review list instead.
What I learned
In bulk work, reversibility beats speed. "Deactivate, don't delete" turned a lot of potential mistakes into harmless ones.
Stack
- Python
- PostgreSQL
- Bedrock