Ferhat RudvanoğullarıData Engineer
← All work

Data quality & master data

Entity Resolution

Matched duplicate company records across 1.2 million rows with tiered rules, from exact website matches down to semantic similarity, with a human in the loop for anything uncertain.

Data engineer, solo

Apr 2026

In numbers

company records
1.2M
company records
matched automatically in the first tier (test set)
96%
matched automatically in the first tier (test set)
shared a website in the first 10K rows
760
shared a website in the first 10K rows

The problem

The same company showed up many times from different sources: "Robert Bosch GmbH", "Bosch", "bosch.de". Buyers saw the same supplier more than once, and its details were scattered across the copies.

What I did

  1. 1

    Make the data matchable

    Raw records lived nested inside a JSONB column. I pulled the fields used for matching (domain, name, country) into their own indexed columns so comparisons over 1.2 million rows stayed fast.

  2. 2

    Three tiers, from certain to uncertain

    Same website means same company, merged automatically. Next, PostgreSQL's trigram matching compares names in three-letter chunks: above 85% similarity in the same country it merges, across countries it goes to review. Finally, names that differ in spelling but mean the same thing are compared as embeddings and always go to review.

  3. 3

    Drop the tier you can't trust

    I started with official company IDs as the most trusted tier. They turned out to be patchy and inconsistent, so I removed that tier entirely.

  4. 4

    A review screen for the rest

    Uncertain matches land in a review screen grouped by domain, where a person merges or rejects them in one click.

Result

On the test set, 96% of records matched automatically in the first tier, 105 more were grouped by name similarity, and the rest went to the review queue.

What I learned

An automatic merge is hard to undo. Automate what is certain, send what is not to a person, and the system stays trustworthy.

Stack

  • PostgreSQL (pg_trgm, JSONB)
  • Embeddings
  • TypeScript
  • React