Ferhat RudvanoğullarıData Engineer
← All work

Data pipelines

Lead Generation & Website Scoring

Built an 11-step pipeline for Vidyon that finds German solar installers, pulls their contact details with an LLM and scores every website on PageSpeed.

Data engineer, solo project for Vidyon

Jun–Jul 2026

In numbers

target companies, 98.3% in the right segment
1,979
target companies, 98.3% in the right segment
phone numbers and 2,941 emails
3,658
phone numbers and 2,941 emails
sites scored on mobile and desktop
9,780
sites scored on mobile and desktop
average mobile performance score
61.5
average mobile performance score

The problem

Vidyon wanted to sell web and video services to solar panel installers in Germany, but there was no clean list of those companies. The sales team was googling firms one at a time and copying contact details by hand.

What I did

  1. 1

    Collect and clean

    I gathered candidate sites through Jina's search API from automatically generated queries. When the API hit its billing limit, the collector moved on to a fresh query instead of retrying the same one. After reducing everything to root domains and removing duplicates, 9,757 sites were left.

  2. 2

    Crawl with fallbacks

    I crawled every site's title, description, headings and structured data with 150 concurrent connections. Sites that wouldn't load fell back to the Common Crawl archive, then to Jina's reader.

  3. 3

    Installer or online shop?

    Llama 3.1 8B on Bedrock labelled each site as an installer or a parts shop. I also embedded every site with Amazon Titan and measured how close it was to a known good example. 33 shops were dropped.

  4. 4

    Extract the leads

    I found each company's contact and imprint pages through DNS records and Athena, pulled their content from Common Crawl with byte-range requests, and had Llama 3.1 extract phone numbers, emails, addresses and social profiles. Phone numbers were normalised to international format and de-duplicated.

  5. 5

    Score every website

    Google's PageSpeed API scored every site for performance, SEO, accessibility and best practices, separately on mobile and desktop. Everything moved from local SQLite to Turso, and I wrote a short report for the sales team.

Result

The sales team got a ready-to-use list in which 76–82% of companies had contact details. The low mobile scores, averaging 61.5, turned into a concrete opening line: "Your site is slow on phones."

What I learned

A data product is only worth what its users can act on straight away: not a raw list, but something to say when they pick up the phone.

Stack

  • Python · HTTPX
  • Jina
  • Bedrock (Llama 3.1 8B, Titan)
  • Athena
  • Common Crawl
  • PageSpeed API
  • Turso