Senior Principal AI Software Engineer (Generative AI Applications Architect, Network Security)

Role Overview

Join Palo Alto Networks as a Senior Principal AI Software Engineer focused on Generative AI applications and agentic systems. You will architect and build AI-native, enterprise-grade solutions, lead retrieval pipeline and AI search experience design, and partner across teams to drive systems into production.

Responsibilities

• Architect and build enterprise-grade Generative AI applications combining LLMs, retrieval, structured data access, search, and orchestration/workflow automation.
• Design agentic systems and multi-step reasoning workflows with state control, memory, tool invocation, and human-in-the-loop checkpoints (e.g., LangGraph or equivalent).
• Lead retrieval pipeline design including:
  • lexical, semantic, and vector search
  • metadata filtering
  • reranking
  • hybrid retrieval strategies
• Deliver distributed, cloud-native solutions with strong reliability, observability, and performance practices.
• Implement AI evaluation frameworks (offline benchmarking, regression testing, trace analysis, quality scoring, and production KPI measurement).
• Build/implement LLM guardrails and safety controls, including policy-based filtering, grounding validation, boundary handling, and safe tool invocation.
• Collaborate with senior engineers, architects, product leaders, and business stakeholders to influence technical direction and execution.

Requirements

• 10+ years building large-scale distributed systems, platforms, or enterprise applications (hands-on depth).
• Strong knowledge of distributed systems, cloud-native architecture, reliability engineering, observability, and performance optimization.
• Proven experience shipping production AI/LLM applications, including agentic workflows, RAG systems, and intelligent automation.
• Deep experience with vector stores, embeddings, and retrieval/semantic search architectures.
• Experience designing reasoning systems combining structured + unstructured data, including patterns such as text-to-SQL, retrieval-augmented workflows, API-based reasoning, and workflow decision systems.
• Strong experience with agent orchestration frameworks such as LangGraph (or equivalent stateful approaches).
• Strong Python skills plus at least one additional language such as Go or Java.
• Experience with Temporal (or similar durable workflow orchestration) for long-running, reliable execution.
• Strong experience with search platforms such as Elasticsearch or OpenSearch (indexing, relevance tuning, filtering, full-text retrieval, and hybrid search design).
• Ability to thrive in ambiguity, rapidly evaluate new technologies, and move from concept to production.

Nice-to-Haves

• Experience building vertical AI applications for domains such as cybersecurity, IT operations, customer support, sales, finance, legal, or enterprise productivity.
• Experience with AI observability platforms for prompt/workflow tracing and operational debugging.
• Experience building shared AI infrastructure/internal AI platforms for multiple teams.
• Familiarity with cloud-native systems (partially truncated in source).