Open source
Company Deduplication
Tried five different approaches to finding duplicate companies in 477,600 records, and wrote down why each one was dropped.
Personal open-source project
2026
In numbers
- raw records, whole pipeline in ~8 min
- 477,600
- raw records, whole pipeline in ~8 min
- candidate pairs after the first pass
- 1.1M
- candidate pairs after the first pass
- name variants down to parent companies
- 1,166 → 142
- name variants down to parent companies
- companies matched to an official website
- 80%+
- companies matched to an official website
The problem
In company data pulled from several sources, the same company showed up under dozens of spellings: "Bosch GmbH", "GmbH Bosch", "Robert Bosch Ltd." There was no getting a clean company list without merging them.
What I did
- 1
Rules (Splink)
Company data had so many exceptions that tuning the rules by hand stopped being practical.
- 2
Brute force (DuckDB)
Comparing every record with every other record grows quadratically. At 477,000 rows it ran out of memory.
- 3
Vector search (FAISS)
I turned names into 384-dimensional embeddings and searched for neighbours. Fast, but unstable on Apple Silicon, and it kept crashing.
- 4
Spatial tree (BallTree)
Same embeddings, searched with a tree structure instead. The first pass produced 1.1 million candidate pairs.
- 5
Fuzzy clustering (PolyFuzz + RapidFuzz)
A comparison that ignores word order ("Bosch GmbH" = "GmbH Bosch"), plus normalisation that strips legal suffixes like GmbH, Ltd and Inc, produced the final clusters. Records sharing the same official website were then merged as well.
Result
477,600 rows were processed in about 8 minutes and collapsed into 142 parent companies (from 1,166 distinct name spellings), and more than 80% of them were matched to an official website. The README explains why each approach was dropped.
What I learned
The best algorithm is the one that actually finishes on your hardware, at your data size.
Stack
- Python
- DuckDB
- Sentence-Transformers
- FAISS
- scikit-learn
- PolyFuzz
- RapidFuzz