Senior Kubernetes Engineer

Role Overview

NorthMark is seeking a Senior Kubernetes Engineer in Dallas to design, implement, and optimize GPU-accelerated Kubernetes container platforms for AI/ML, HPC, and LLM training workloads across hybrid or on-prem environments.

Responsibilities

• Architect and operate GPU-optimized Kubernetes clusters using NVIDIA tooling such as:
  • NVIDIA GPU Operator, Network Operator, and DCGM
• Build and maintain custom Kubernetes operators and controllers to automate infrastructure services
• Integrate NVIDIA GPU capabilities into scheduling, including:
  • Device plugins, MIG (Multi-Instance GPU), and GPU sharing
• Improve GPU utilization and placement via scheduler extensions and integrations such as:
  • kube-scheduler plugins, Slurm, Volcano
• Partner with HPC, ML, and DevOps teams for multi-tenant, high-throughput cluster performance
• Drive observability and telemetry using:
  • Prometheus, Grafana, DCGM Exporter, OpenTelemetry
• Implement secure multi-user/multi-namespace isolation using RBAC and policy tools like:
  • OPA or Gatekeeper
• Maintain Kubernetes infrastructure CI/CD with GitOps workflows using ArgoCD and FluxCD
• Contribute to Infrastructure as Code using Terraform, Helm, and Kustomize
• Participate in performance tuning, incident response, and production readiness reviews

Requirements

• Extensive production Kubernetes experience, including with NVIDIA GPU ecosystems (e.g., GPU Operator, device plugins, NVML, MIG, DCGM)
• Proficiency in Go or Python for operator and controller development
• Deep understanding of Kubernetes internals including CRDs, RBAC, custom controllers, and scheduler extensions
• Hands-on experience with GPU-intensive workloads (LLMs, training pipelines, scientific computing)
• Experience with Helm, Kustomize, and GitOps workflows
• Familiarity with CNI plugins, especially NVIDIA CNI and Multus
• Monitoring experience using Prometheus and DCGM Exporter

Benefits (highlights)

• Company-paid lunch stipend
• 100% employer-paid medical (High Deductible Health Plan) + dental and vision
• 16 weeks paid parental leave
• EAP, life insurance, short/long-term disability
• 401(k) match: 100% up to 6%