Cloud Inference Engineer

Role overview

Join Modular’s Cloud Inference team to build and ship an LLM-focused inference platform. You’ll help push operational excellence for distributed inference systems while collaborating with the company’s kernel and genAI teams.

Key missions

• Build and deliver an LLM inference platform using best-in-class inference techniques.
• Collaborate with kernels and genAI teams to reach state-of-the-art application performance.
• Create Helm charts, Kubernetes operators, and other components to enable simple, efficient, maintainable deployments.

Responsibilities

• Develop and productionize inference infrastructure and deployment tooling.
• Work across large-scale backend systems and ML inference.
• Build durable reusable software tools/libraries leveraged across teams.

Requirements

• Strong backend engineering breadth/depth with experience in AI inference and distributed systems.
• 5+ years backend engineering experience.
• Experience with machine learning technologies and use cases.
• Kubernetes experience, including operating your own services.
• Ability to create durable reusable software tools/libraries.
• Experience building high-scale ML inference infrastructure (traditional AI or genAI).
• Familiarity with Go (Golang).

Nice-to-haves

• Interest and experience with high performance computing / networking.
• Team-oriented attitude and alignment with company culture/values.