Systems I build and run. Each one is in production, and most of what I write about comes from them.
Data Department B.V.
My company · Platform, data products and compliance
Data Department collects public data from Dutch websites, such as company information, job postings and real estate listings, and turns it into datasets and data products for the Dutch and EU market.
The problem
Websites change without warning, block automated visitors and hide their data behind JavaScript. The data still has to arrive every day, complete and collected within the law.
What I built
- A scraping platform on about 30 Leaseweb servers: Scrapy spiders share Redis task queues, and Nomad decides which server runs what.
- A Playwright setup that solves cookie walls and bot checks once, so the fast scrapers can reuse the result.
- Collectors for single-page apps, found by reverse engineering their APIs with mitmproxy and scrapy-impersonate.
- ClickHouse for analytics on everything collected, with PostgreSQL as the operational database.
- A tiered set of data products built on top.
- GDPR/AVG compliance work for B2B data, including analysis of the CJEU KNLTB ruling and changes to the Dutch Handelsregisterwet.
Stack: Scrapy · Redis · Nomad · Playwright · mitmproxy · scrapy‑impersonate · ClickHouse · PostgreSQL
WPOptic
Product · Scraping pipeline, Chrome extension and frontend integration
WPOptic detects WordPress and the technology behind websites.
How it is built
WPOptic runs on a ClickHouse backend with a Go API and a FastAPI layer. I built three parts of it:
- The scraping pipeline that collects the websites to analyse.
- A Chrome extension that shows the results for the site you are visiting.
- The connection between the WordPress frontend and the backend APIs.
Stack: ClickHouse · Go · FastAPI · Chrome extension · WordPress
Need something like this?
I build systems like these for other companies too: data feeds, pipelines and AI tooling that run on infrastructure you control. See services → or book a free data consult.