Ferhat RudvanoğullarıData Engineer
← All work

Data quality & master data

Address Normalization

Replaced a paid AI call per address with a self-hosted ML address parser (libpostal) and took throughput from 100 to over 15,000 addresses a minute.

Data engineer, solo

Apr 2026

In numbers

addresses per minute
100 → 15,000+
addresses per minute
est., 1.07M addresses
~35 h → ~1.5 h
est., 1.07M addresses
database commands per 1,000 rows
1,000 → 3
database commands per 1,000 rows
model cost per address
$0
model cost per address

The problem

Supplier addresses were stored as a single line of free text, like "Industriestrasse 11, 63877 Sailauf, Bayern". With no separate street, city or postcode fields, you couldn't search suppliers by city or region. On top of that, every address was being sent to a paid AI service to be split up, one call at a time. For a million addresses that was slow and expensive.

What I did

  1. 1

    Our own address model instead of a paid AI call

    libpostal is an open-source machine learning model, trained on OpenStreetMap data, that splits addresses from almost anywhere into street, number, city, postcode and country. I packaged it in a Docker image that compiles the library and bundles its trained data files (a few GB; the build step alone takes 20–30 minutes), put a small Node.js + Hono HTTP service in front of it, and wrote an ETL script that reads addresses from the database, sends them to the service, writes the results back and keeps re-running until nothing is pending. Cost per address dropped to zero. The catch: the first run only handled 100 addresses a minute, which meant about a week for the full dataset.

  2. 2

    Parallel requests, keyset reads

    I started sending addresses concurrently instead of one after another, and switched the database reads from OFFSET to keyset pagination so they stopped slowing down as the job progressed. That got us to 500 a minute, five times faster, but still around 35 hours in total.

  3. 3

    Use the whole CPU

    Watching the server, I noticed only one of its 8 threads was busy, at 27%. A single model instance was barely touching the CPU. I ran the parser as 4, then 8 instances and spread requests across them, for roughly 4× the speed of a single instance.

  4. 4

    Fix the real bottleneck: writes

    Now the parser was fast, but every result was still written with its own UPDATE: 1,000 commands per 1,000 rows. I collected results in memory and wrote each batch in one go (temp table + COPY, then a single bulk update), turning 1,000 commands into 3.

Result

Throughput went from 100 to over 15,000 addresses a minute, roughly 150×. The estimate for a million addresses fell from about 35 hours to under an hour and a half, the per-address AI cost disappeared, and suppliers became searchable by city, region and postcode.

How it works

  1. PostgreSQL

    Batches of 500, read with keyset pagination

  2. ETL script (TypeScript)

    50 requests in flight at once

  3. libpostal API × 8

    Docker, Node.js + Hono, one instance per CPU thread

  4. Results buffered in memory

    COPY into a temp table, then one bulk UPDATE

  5. PostgreSQL

    Street, number, city, postcode and country in their own columns

What I learned

The bottleneck moved every time I fixed one: cost, then sequential requests, then CPU, then the database. Measuring at each step, instead of guessing, is what pointed at the right fix.

Stack

  • libpostal
  • Docker
  • Node.js + Hono
  • TypeScript (Bun)
  • PostgreSQL