Data pipelines
Lead Generation & Website Scoring
Built an 11-step pipeline for Vidyon that finds German solar installers, pulls their contact details with an LLM and scores every website on PageSpeed.
Data engineer, solo project for Vidyon
Jun–Jul 2026
In numbers
- target companies, 98.3% in the right segment
- 1,979
- target companies, 98.3% in the right segment
- phone numbers and 2,941 emails
- 3,658
- phone numbers and 2,941 emails
- sites scored on mobile and desktop
- 9,780
- sites scored on mobile and desktop
- average mobile performance score
- 61.5
- average mobile performance score
The problem
Vidyon wanted to sell web and video services to solar panel installers in Germany, but there was no clean list of those companies. The sales team was googling firms one at a time and copying contact details by hand.
What I did
- 1
Collect and clean
I gathered candidate sites through Jina's search API from automatically generated queries. When the API hit its billing limit, the collector moved on to a fresh query instead of retrying the same one. After reducing everything to root domains and removing duplicates, 9,757 sites were left.
- 2
Crawl with fallbacks
I crawled every site's title, description, headings and structured data with 150 concurrent connections. Sites that wouldn't load fell back to the Common Crawl archive, then to Jina's reader.
- 3
Installer or online shop?
Llama 3.1 8B on Bedrock labelled each site as an installer or a parts shop. I also embedded every site with Amazon Titan and measured how close it was to a known good example. 33 shops were dropped.
- 4
Extract the leads
I found each company's contact and imprint pages through DNS records and Athena, pulled their content from Common Crawl with byte-range requests, and had Llama 3.1 extract phone numbers, emails, addresses and social profiles. Phone numbers were normalised to international format and de-duplicated.
- 5
Score every website
Google's PageSpeed API scored every site for performance, SEO, accessibility and best practices, separately on mobile and desktop. Everything moved from local SQLite to Turso, and I wrote a short report for the sales team.
Result
The sales team got a ready-to-use list in which 76–82% of companies had contact details. The low mobile scores, averaging 61.5, turned into a concrete opening line: "Your site is slow on phones."
What I learned
A data product is only worth what its users can act on straight away: not a raw list, but something to say when they pick up the phone.
Stack
- Python · HTTPX
- Jina
- Bedrock (Llama 3.1 8B, Titan)
- Athena
- Common Crawl
- PageSpeed API
- Turso