Ferhat RudvanoğullarıData Engineer
← All work

Open source

Company Deduplication

Tried five different approaches to finding duplicate companies in 477,600 records, and wrote down why each one was dropped.

Personal open-source project

2026

In numbers

raw records, whole pipeline in ~8 min
477,600
raw records, whole pipeline in ~8 min
candidate pairs after the first pass
1.1M
candidate pairs after the first pass
name variants down to parent companies
1,166 → 142
name variants down to parent companies
companies matched to an official website
80%+
companies matched to an official website

The problem

In company data pulled from several sources, the same company showed up under dozens of spellings: "Bosch GmbH", "GmbH Bosch", "Robert Bosch Ltd." There was no getting a clean company list without merging them.

What I did

  1. 1

    Rules (Splink)

    Company data had so many exceptions that tuning the rules by hand stopped being practical.

  2. 2

    Brute force (DuckDB)

    Comparing every record with every other record grows quadratically. At 477,000 rows it ran out of memory.

  3. 3

    Vector search (FAISS)

    I turned names into 384-dimensional embeddings and searched for neighbours. Fast, but unstable on Apple Silicon, and it kept crashing.

  4. 4

    Spatial tree (BallTree)

    Same embeddings, searched with a tree structure instead. The first pass produced 1.1 million candidate pairs.

  5. 5

    Fuzzy clustering (PolyFuzz + RapidFuzz)

    A comparison that ignores word order ("Bosch GmbH" = "GmbH Bosch"), plus normalisation that strips legal suffixes like GmbH, Ltd and Inc, produced the final clusters. Records sharing the same official website were then merged as well.

Result

477,600 rows were processed in about 8 minutes and collapsed into 142 parent companies (from 1,166 distinct name spellings), and more than 80% of them were matched to an official website. The README explains why each approach was dropped.

What I learned

The best algorithm is the one that actually finishes on your hardware, at your data size.

Stack

  • Python
  • DuckDB
  • Sentence-Transformers
  • FAISS
  • scikit-learn
  • PolyFuzz
  • RapidFuzz
Code on GitHub