Ferhat RudvanoğullarıData Engineer
← All work

Data pipelines

Supplier Page Classification

Located half a million suppliers' websites in the Common Crawl archive and classified their pages into seven categories: 485,562 domains in under 25 minutes.

Data engineer, solo

Jun 2026

In numbers

domains in 24m 43s
485,562
domains in 24m 43s
pages in 7 categories
1.08M
pages in 7 categories
handled by rules / by an LLM
97% / 3%
handled by rules / by an LLM
pages fetched from the archive
124/s
pages fetched from the archive

The problem

SupplyBridge had over half a million suppliers on file, but for most of them we didn't know what they made, which certifications they held or where they produced. That information sat on their own websites. Crawling half a million sites one by one would have taken weeks and run straight into rate limits and blocks.

What I did

  1. 1

    Query the archive instead of crawling

    Common Crawl is a public archive of a large slice of the web, refreshed monthly and stored on AWS S3. It's petabytes in size, so downloading it is out of the question, but it ships with an index that records where every page lives. I loaded our supplier list into S3 and joined it against that index with Athena, which runs SQL straight over files in S3. The query took 260 seconds and gave me the exact location of every page we needed. 358,590 suppliers turned up in the archive.

  2. 2

    Fetch only the bytes you need

    Each archive file is roughly 1 GB. Rather than pulling whole files, I used S3 range requests to fetch just the 10–30 KB slice for each page. Spread across 40 parallel workers, that came to 124 pages a second.

  3. 3

    Cheap rules first, an LLM for the rest

    Only a handful of pages on a company site really matter: home, about, products, certifications, capabilities, locations and contact. Multilingual URL and title patterns caught 1,046,001 of them. Whatever the rules couldn't place went to a small, cheap model on Bedrock (Amazon Nova Lite), which found another 37,404. Results were written back in batches of 5,000.

Result

1.08 million pages from 485,562 domains were classified in under 25 minutes, about 327 domains a second. The LLM only touched 3% of the work, which kept both cost and runtime small. The output became the input for the next stage: turning these pages into supplier profiles.

How it works

  1. Supplier domains → S3

    Our supplier list, uploaded as a table

  2. Athena

    Joined with the Common Crawl index in 260 s: file and byte range for every page

  3. S3 range requests

    40 workers, 10–30 KB per page, 124 pages/s

  4. Multilingual URL and title rules

    1,046,001 pages classified; the rest move on

  5. Bedrock (Nova Lite)

    37,404 pages the rules couldn't place

  6. PostgreSQL

    Written in batches of 5,000

What I learned

At this scale the cheapest request is the one you never send. Keeping the LLM for the small slice the rules couldn't handle cut both time and cost.

Stack

  • Common Crawl
  • AWS Athena
  • S3
  • Bedrock (Nova Lite)
  • Python
  • PostgreSQL