ML Platform Engineer

Role Overview

Bright Vision Technologies is hiring a ML Platform Engineer to design, build, and operate high-performance, highly reliable inference platforms for serving large machine learning models in production. This role focuses on the systems engineering side of AI deployment, including request routing, batching, caching, autoscaling, GPU utilization, and end-to-end observability.

Responsibilities

• Design and operate model serving platforms for diverse workloads (LLMs, vision models, recommendation systems)
• Optimize inference performance using techniques such as:
  • continuous batching
  • paged attention
  • speculative decoding
  • request multiplexing
• Implement multi-tenant routing, rate limiting, and quality-of-service (QoS) policies across model endpoints
• Build autoscaling and capacity management systems balancing latency, throughput, and cost
• Tune GPU utilization and memory management, including KV cache strategies for LLM workloads
• Integrate model serving with API gateways, identity systems, and observability platforms
• Apply caching, prompt deduplication, and response reuse strategies where appropriate
• Deliver end-to-end observability (latency histograms, queue dynamics, GPU utilization, error tracking)
• Develop deployment workflows such as canary releases, shadow testing, and automated rollback
• Operate incident response for high-availability AI services and drive reliability improvements
• Collaborate with ML and product teams to support new model releases

Requirements

• 6+ years of experience
• Strong distributed systems and performance engineering expertise
• Demonstrated experience shipping ML serving systems at scale
• Understanding of trade-offs between latency, throughput, cost, and quality for ML serving
• Must complete a mandatory technical coding assessment

Nice-to-Haves

• Experience with advanced ML serving techniques (e.g., continuous batching, paged attention, speculative decoding)
• Experience integrating observability and identity/API gateway systems

Employment / Logistics

• Type: Full-time, direct W2 employee (no C2C/1099/third-party)
• Engagement: In-house Bright Vision Technologies SOW engagement (long-term, multi-year)
• Remote: 100% remote (Continental United States)
• Visa: No new H1B sponsorship; H1B transfers welcomed
• Compensation: $100k–$150k base + benefits