Summary

Build web-scale crawling and content acquisition infrastructure for Nebius's agentic search platform. The role focuses on distributed systems that discover, fetch, refresh, and prepare real-world data for AI search, with the working arrangement not specified.

Responsibilities

• Design and operate web-scale crawling systems
• Build ingestion workflows for diverse data sources
• Develop crawl scheduling and recrawl strategies
• Implement URL discovery and content extraction
• Ensure reliable high-throughput infrastructure operation
• Define crawl coverage and freshness metrics
• Monitor resource usage and infrastructure costs