Senior Staff Software Engineer, High Performance Inference System

About Groq Groq delivers fast, efficient AI inference. Our LPU-based system powers GroqCloud™, giving businesses and developers the speed and scale they need. Headquartered in Silicon Valley, we are on a mission to make high performance AI compute more accessible and affordable. When real-time AI is within reach, anything is possible. Build fast. Senior Staff Software Engineer – High Performance Inference System Mission: Join the team that builds and operates Groq’s real-time, distributed inference system delivering large scale inference for LLMs and next-gen AI applications at ultra-low latency. Your work will optimize for heterogeneous hardware, dynamic global workloads, and extreme performance—all while running code at the edge of physics. Responsibilities & opportunities in this role:

• Distributed Systems Engineering: Design and implement scalable, low-latency runtime systems that coordinate thousands of GroqChips across a software-scheduled interconnect.
• Low-Level Optimization: Develop deterministic, hardware-aware abstractions that prioritize execution speed, fault tolerance, and reliability.
• Performance & Diagnostics: Build tools and infrastructure to support real-time system observability, diagnostics, and SLO improvements.
• Future-Proofing: Evolve Groq’s system stack to support emerging silicon, topologies, and heterogeneous accelerators (e.g., FPGAs).
• Cross-Functional Collaboration: Partner with teams across compiler, infra, cloud, hardware, and data centers to align architecture and drive shared progress.

Ideal candidates have/are:

• Consistently ship high-impact, production-ready systems code.
• Have deep knowledge of computer architecture, operating systems, algorithms, and hardware-software interfaces.
• Are fluent in low-level systems languages such as C++ or Rust, and comfortable with hardware-aware programming.
• Rigorously profile and optimize for latency, throughput, and resource efficiency—every cycle counts.
• Believe in automation and CI/CD best practices—you don’t ship untested code.
• Thrive across the stack—from kernel internals to hardware integration to cloud load balancers.
• Communicate clearly, make pragmatic technical decisions, and write maintainable code for the long term.
• Ensures code stays fast, scales well, and takes ownership of outcomes.

Nice to have:

• Operating large-scale distributed systems for real-time, high-traffic services.
• Deploying and optimizing ML or HPC workloads in production environments.
• Hands-on experience with GPUs, FPGAs, or ASICs in performance-critical systems.
• Familiarity with ML frameworks (e.g., PyTorch) or compiler tools (e.g., MLIR).
• Experience delivering complex projects in fast-paced, high-impact environments.

Attributes of a Groqster:

• Humility - Egos are checked at the door
• Collaborative & Team Savvy - We make up the smartest person in the room, together
• Growth & Giver Mindset - Learn it all versus know it all, we share knowledge generously
• Curious & Innovative - Take a creative approach to projects, problems, and design
• Passion, Grit, & Boldness - no limit thinking, fueling informed risk taking

If this sounds like you, we’d love to hear from you! Compensation: At Groq, a competitive base salary is part of our comprehensive compensation package, which includes equity and benefits. For this role, the base salary range is $248,710 - $336,490, determined by your skills, qualifications, experience and internal benchmarks. Location: Some roles may require being located near or on our primary sites, as indicated in the job description. At Groq: Our goal is to hire and promote an exceptional workforce as diverse as the global populations we serve. Groq is an equal opportunity employer committed to diversity, inclusion, and belonging in all aspects of our organization. We value and celebrate diversity in thought, beliefs, talent, expression, and backgrounds. We know that our individual differences make us better. Groq is an Equal Opportunity Employer that is committed to inclusion and diversity. Qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, gender, sexual orientation, gender identity, disability or protected veteran status. We also take affirmative action to offer employment opportunities to minorities, women, individuals with disabilities, and protected veterans. Groq is committed to working with qualified individuals with physical or mental disabilities. Applicants who would like to contact us regarding the accessibility of our website or who need special assistance or a reasonable accommodation for any part of the application or hiring process may contact us at: [email protected]. This contact information is for accommodation requests only. Evaluation of requests for reasonable accommodations will be determined on a case-by-case basis. Apply tot his job