Data pipelines
Supplier Page Classification
Located half a million suppliers' websites in the Common Crawl archive and classified their pages into seven categories: 485,562 domains in under 25 minutes.
Data engineer, solo
Jun 2026
In numbers
- domains in 24m 43s
- 485,562
- domains in 24m 43s
- pages in 7 categories
- 1.08M
- pages in 7 categories
- handled by rules / by an LLM
- 97% / 3%
- handled by rules / by an LLM
- pages fetched from the archive
- 124/s
- pages fetched from the archive
The problem
SupplyBridge had over half a million suppliers on file, but for most of them we didn't know what they made, which certifications they held or where they produced. That information sat on their own websites. Crawling half a million sites one by one would have taken weeks and run straight into rate limits and blocks.
What I did
- 1
Query the archive instead of crawling
Common Crawl is a public archive of a large slice of the web, refreshed monthly and stored on AWS S3. It's petabytes in size, so downloading it is out of the question, but it ships with an index that records where every page lives. I loaded our supplier list into S3 and joined it against that index with Athena, which runs SQL straight over files in S3. The query took 260 seconds and gave me the exact location of every page we needed. 358,590 suppliers turned up in the archive.
- 2
Fetch only the bytes you need
Each archive file is roughly 1 GB. Rather than pulling whole files, I used S3 range requests to fetch just the 10–30 KB slice for each page. Spread across 40 parallel workers, that came to 124 pages a second.
- 3
Cheap rules first, an LLM for the rest
Only a handful of pages on a company site really matter: home, about, products, certifications, capabilities, locations and contact. Multilingual URL and title patterns caught 1,046,001 of them. Whatever the rules couldn't place went to a small, cheap model on Bedrock (Amazon Nova Lite), which found another 37,404. Results were written back in batches of 5,000.
Result
1.08 million pages from 485,562 domains were classified in under 25 minutes, about 327 domains a second. The LLM only touched 3% of the work, which kept both cost and runtime small. The output became the input for the next stage: turning these pages into supplier profiles.
How it works
Supplier domains → S3
Our supplier list, uploaded as a table
Athena
Joined with the Common Crawl index in 260 s: file and byte range for every page
S3 range requests
40 workers, 10–30 KB per page, 124 pages/s
Multilingual URL and title rules
1,046,001 pages classified; the rest move on
Bedrock (Nova Lite)
37,404 pages the rules couldn't place
PostgreSQL
Written in batches of 5,000
What I learned
At this scale the cheapest request is the one you never send. Keeping the LLM for the small slice the rules couldn't handle cut both time and cost.
Stack
- Common Crawl
- AWS Athena
- S3
- Bedrock (Nova Lite)
- Python
- PostgreSQL