Location
Anywhere (US-EU timezones)
Job posted
November 25, 2025
Type
⚙️ Engineering
We’re a fast-growing Seed-stage startup building the next-generation web search engine designed for AI and LLM applications. We are building a product which will operate on clusters of 1,000+ machines, processing petabytes of data and powering large-scale retrieval for production AI workloads.
This is a rare opportunity to own core infrastructure end-to-end and shape how AI systems interact with the modern web.
Design, develop, and maintain scalable a distributed crawling infrastructure that powers our entire search and indexing pipeline.
Collaborate on indexing, vectorization, and storage systems driven by the crawler .
Ensure operational excellence, observability, and system reliability.
Work closely with other team members to develop, test, and deploy new features and enhancements for our platform.
Troubleshoot, debug, and optimize performance.
Proven experience building distributed crawling systems that processed several millions of pages per day.
Strong proficiency in Rust or another high-performance systems language.
Deep understanding of scalable content-fetching, politeness policies, scheduling, deduplication, and URL management.
Familiarity with handling modern web complexity: dynamic pages, client-side rendering, rate limits, anti-bot challenges.
Strong communication and interpersonal skills, with the ability to collaborate effectively with both technical and non-technical team members.
Experience with cloud platforms (e.g., AWS, GCP, Azure).
Familiarity with containerization technologies (e.g., Docker, Kubernetes).
Previous experience working in a startup environment.
Experience with Python for pipeline/ML integration
Experience with JS-enabled crawling , headless browsers, or rendering frameworks
Competitive salary and equity package.
Opportunity to make a meaningful impact in a fast-growing startup.
A supportive, collaborative, and diverse work environment.
Flexible work hours and remote work options (anywhere compatible with US-EU timezones).
Professional development and continuous learning opportunities.
Comprehensive benefits package, including US health insurance, US 401k , flexible time off and more .
We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3).
If you are a Web Crawler Engineer with a passion for designing and building distributed crawling infrastructures we'd love to hear from you. Apply now and join our mission to revolutionize web search!
Seltz is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
Email your application jobs@seltz.ai
...Aerotek - JobID: JP-005853505 [Warehouse Associate / Forklift Operator] As a Reach Operator at Aerotek, you'll: Perform loading, unloading, put-away, and stocking of products received at locations; Use the reach truck & cherry picker; Match SKU by cycle counting; Consolidate...
...will be part of a team that contributes to a culture that positively impacts the communities and customers we serve. Job Description As an Airport Security Officer: Patrol Response in Kailua Kona, HI , you will serve and safeguard clients in a range of...
...Job Description Job Description About Blue Grass Airport: At Blue Grass Airport (LEX), our employees are the heart of our operation... ...every day. Position Summary: Sworn law enforcement officers perform the traditional Police Officer responsibilities in a unique...
...Join Us as Grant Writer to Act as Our Lead Expert Reviewer! We're looking for an exceptionally thoughtful, detail-driven Grant Delivery Specialist to lead high-quality grant research, writing and review for our nonprofit partners. You'll help nonprofits access more...
Columbus Specialty Hospital of Newark, NJ is seeking to hire a full time Receptionist to handle the main lobby reception desk for our long-term acute care hospital. We are looking for a Receptionist to work Monday thru Friday, 8 hour shifts. &n... Silver Lake Hospital