Summary

Nebius is hiring a staff software engineer to build web-scale crawling and content acquisition infrastructure for its agent-native search platform. The role focuses on distributed systems that discover, refresh, and ingest web content for AI retrieval.

Responsibilities

• Design web-scale crawling systems
• Build ingestion workflows for data sources
• Develop crawl scheduling and freshness policies
• Implement URL discovery and content extraction
• Define crawl coverage and freshness metrics
• Monitor bandwidth usage and infrastructure costs