Staff Backend Engineer

Role overview

Staff Backend Engineer at Grafana Labs (remote-first). You’ll build and operate backend systems that power Grafana Cloud capabilities such as the billing engine, customer provisioning, and the user portal—especially around reconciliation/state management and reliable lifecycle workflows.

Key missions

• Design, build, and operate reconciliation systems to track desired stack state and detect/repair drift across stack templates.
• Collaborate across teams to keep stack lifecycle workflows reliable, observable, and resilient, improving operational efficiency.
• Manage rollout mechanisms for provisioned plugins, dashboards, data sources, Grafana versions, release channels, and stack-level configuration.
• Contribute to roadmap planning, technical design, and incident response.

What you’ll be great at

• Building production systems where correctness, scalability, and operational clarity are critical.
• Turning ambiguous lifecycle requirements into explicit, modular solutions.
• Working comfortably with stateful systems, eventual consistency, and reconciliation loops.
• Breaking down complex work into safe, iterative increments and communicating tradeoffs to internal stakeholders and product teams.

Requirements

• Professional experience with Go (Golang); comfortable working across backend service and application code.
• Strong software engineering habits: clean, robust, well-tested code.
• Experience delivering projects end-to-end (from requirements/discovery to shipping).
• Experience with blameless incident response and writing high-quality post-incident reviews.
• Experience on large SaaS platforms and distributed systems challenges (scalability, multi-tenancy, data isolation, HA, etc.).
• Ability to lead major projects in a small, independent team and coordinate across service boundaries.
• Kubernetes experience in AWS, GCP, or Azure, plus familiarity with infrastructure-as-code (Helm, Terraform, Jsonnet, etc.).
• At least 1 year of fully remote work experience.
• Some experience with TypeScript/Node.js.

Nice-to-haves / preferred experience

• Kubernetes control-plane patterns, operators, reconcilers, or desired-state systems.
• Deployment/config tooling such as Jsonnet/Tanka, Terraform, Flux, Argo, or similar.
• Experience with SaaS provisioning, tenancy, regional expansion, plugin rollout, or customer lifecycle systems.
• Incident response experience involving configuration drift.
• Experience mentoring junior engineers in an asynchronous, collaborative environment.
• Care about developer/user experience and product quality.