Research Crawling Engineer

About the Role

Join a data infrastructure team building large-scale web crawling systems and datasets for AI research and model development. This role focuses on reliable data acquisition across diverse, changing web environments.

What You'll Do

• Build and maintain web crawlers across social, travel, and multilingual websites.
• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
• Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy sites.
• Develop pipelines to clean, deduplicate, filter, and normalize web data at TB-to-PB scale.
• Build and maintain datasets for research and model training in collaboration with research teams.
• Monitor crawl performance, coverage, and data quality, and adjust systems as websites change.
• Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.

What We're Looking For

• At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, or 2 or more years with a PhD.
• Hands-on experience building and operating web crawlers at scale, including data pipelines handling TB-scale workloads.
• Strong proficiency in Go, Rust, Python, Java, or C++, with experience in distributed systems or parallel processing.
• Deep knowledge of HTTP, networking, and browser behavior, including experience working with anti-bot systems, rate limits, and JavaScript-heavy sites.
• Experience with headless browsers, proxy systems, IP rotation, and request orchestration is valuable.
• Familiarity with data quality evaluation, large-scale data processing, and cloud or bare-metal infrastructure.
• Experience with NLP pipelines, machine learning dataset curation, or AI research data is a plus.
• A degree in computer science, engineering, or a related technical field is helpful; a PhD is a plus.

Compensation & Benefits

Salary range: $160,000 to $250,000 annually. Visa sponsorship is not available.

Location

Remote.