Senior Data Engineer (Web Scraping)

This is a senior, hands-on opportunity to own the next generation of reliable web-data acquisition within a modern data platform.
You will design, build, and operate production-grade Python scrapers and scalable ingestion pipelines that support trusted research and data products.
The role focuses on improving the maturity, reliability, and scalability of web-scraping capabilities across a growing data environment.
You will establish reusable patterns for extraction, scheduling, storage, monitoring, validation, and failure handling.
From investigating new data sources to deploying and supporting production workloads, you will have substantial ownership over the full engineering lifecycle.
The role combines deep Python engineering with cloud infrastructure, data pipelines, observability, and practical problem-solving in a remote-first environment.
You will work independently while collaborating with a distributed engineering team through code reviews, documentation, and structured development workflows.

• Demonstrated professional experience building and operating production web-scraping systems at scale.
• Proven ability to independently take substantial scraping projects from initial investigation through implementation, deployment, and ongoing production support.
• Strong production-level Python engineering skills, with experience developing maintainable applications rather than standalone scripts.
• Hands-on experience with scraping technologies such as Requests/httpx, BeautifulSoup, Scrapy, Playwright, or Selenium.
• Strong practical understanding of HTTP, HTML, APIs, JavaScript-rendered websites, and browser/network behavior.
• Experience addressing common scraping challenges including pagination, authentication, sessions, retries, rate limiting, concurrency, and proxies.
• Strong understanding of data pipelines, data quality, and how collected data should be validated, stored, and consumed by downstream systems.
• Experience deploying, monitoring, and supporting production workloads in a cloud environment.
• Strong debugging, analytical, and problem-solving abilities, with the judgment to make effective engineering decisions independently.
• Comfortable working within a remote engineering team and participating in code reviews, documentation, and ticket-based development workflows.
• Experience with AWS is desirable.
• Familiarity with lakehouse or data-lake architectures, particularly Apache Iceberg, is a plus.
• Experience with PySpark or other distributed data-processing technologies is beneficial.
• Familiarity with Docker and containerized workloads is advantageous.
• Experience with Terraform or other infrastructure-as-code tools is a plus.
• Familiarity with Grafana or comparable observability platforms is desirable.
• Experience operating high-volume or distributed crawling systems is beneficial.
• Experience evaluating or operating commercial scraping, proxy, or browser-infrastructure services is a plus.
• Experience implementing automated scraper testing, canary runs, or source-drift detection is desirable.
• Exposure to legal, compliance, privacy, or data-governance processes related to web-data acquisition is advantageous.
• Strong ownership, autonomy, documentation, communication, and collaboration skills.