Member of Technical Staff — Training
- Salary
- $200,000 - $400,000
- Level
- Staff
- Work type
- On-site · Palo Alto
Not stated in the posting: visa.
Open. First seen 22 October 2026.
About the role
About the Role
As a Member of Technical Staff, Training, you will design, build, and operate the distributed systems behind large-scale model post-training — spanning training, inference, and orchestration, with a focus on the performance, correctness, scalability, and reliability of workloads running across large GPU clusters.
This role suits engineers who move fluidly across modeling recipes, complex infrastructure, and low-level systems, identify bottlenecks in distributed workloads, and translate experimental requirements into robust software.
In This Role, You Will
- Design, build, and operate distributed training, rollout, and orchestration systems for large-scale LLM and multimodal post-training across multi-GPU, multi-node environments.
- Profile and optimize performance across the full-stack — model implementation, parallelism strategies, communication libraries, and GPU kernels — to improve throughput, latency, memory efficiency, hardware utilization, and cost.
- Investigate numerical correctness and low-precision issues in distributed training and inference, including train–inference consistency for reinforcement learning.
- Improve the reliability of long-running workloads through checkpointing, fault recovery, observability, and operational tooling.
- Build supporting infrastructure for reinforcement learning and agentic post-training, including asynchronous rollout, trajectory collection, sandboxed execution, evaluation harnesses, and data pipelines.
- Contribute to open-source training and inference systems, including Miles and SGLang, and partner with researchers to turn experimental requirements into production systems.
Minimum Qualifications
- 3+ years of experience building or operating distributed machine learning systems, large-scale training infrastructure, or high-performance inference systems.
- Hands-on experience with post-training systems, training backends, or inference systems for large language models (e.g., Megatron-LM, FSDP, SGLang, TensorRT-LLM, vLLM).
- Experience in at least two of the following areas:
- Performance, efficiency, and scalability of multi-GPU, multi-node workloads
- Numerical correctness or low precision
- Stability, reliability, or fault tolerance
- Post-training algorithm recipes and orchestration infrastructure for large training runs
- Multimodal training or inference, including vision-language models and multimodal generation
- Agent infrastructure, including sandboxes, harnesses, and eval systems
- Building and maintaining open-source projects widely adopted in industry and academia
Preferred Qualifications
- Familiarity with RL algorithms such as PPO, GRPO, and their variants, and experience applying them in large-scale post-training.
- Experience with modern post-training frameworks (e.g., Miles, slime, AReaL, verl, Prime-RL).
- Key open-source contributions to training or inference frameworks (e.g., SGLang, vLLM, Megatron-LM).
- GPU kernel development (e.g., CUDA, Triton, CUTLASS) or communication-layer optimization (e.g., NCCL, RDMA, NVLink/NVSwitch).
- Experience training or serving models at very large scale (e.g., Mixture-of-Experts models on clusters of thousands of GPUs).
- Top-tier publications in ML systems or other systems fields.
Even if you don't meet every qualification above, we encourage you to apply — we care most about demonstrated ability to build and reason about large-scale systems.
About RadixArk
RadixArk builds open-source and production infrastructure for large language models and multimodal post-training. Our systems — including Miles, an enterprise-grade reinforcement learning training framework, and SGLang, a widely deployed high-performance LLM inference engine — power distributed post-training across clusters of 10k–100k+ GPUs.
Compensation
Depending on background, skills, and experience, the expected annual salary range for this position is $200,000 to $400,000, plus equity.
Benefits include a 401(k) plan and unlimited PTO.
RadixArk sponsors employment visas (e.g., H-1B, O-1) for eligible candidates.
Equal Opportunity
RadixArk is an Equal Opportunity Employer and is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.
What this role asks for
Particular to this role
Node.js
Rare in RadixArk's other 16 open postings.
In most of RadixArk's ads
LLM
Named in most of its open postings, whatever the role.
Compared with RadixArk's other open jobs.
The team: Member of Technical Staff HIRES STEADILY
This role is one of 14 open in Member of Technical Staff. 1 was posted in the last 30 days and the oldest in December: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
LLM (14 of 14), GitHub (13 of 14), Python (11 of 14), C++ (8 of 14), CUDA (8 of 14), Golang (4 of 14), Machine Learning (4 of 14)
OTHER ROLES ON THIS TEAM
- Member of Technical Staff — Backend/API Platform posted Oct 21
- Member of Technical Staff — Security posted Sep 18
- Member of Technical Staff — Inference-Multi-Hardware posted Aug 3
- Member of Technical Staff — Developer Experience posted Jul 23
- Member of Technical Staff — Developer Technology posted Jul 23
- Member of Technical Staff — Performance posted May 17
- Member of Technical Staff — Product posted May 7
- Member of Technical Staff — Reliability-CI Infrastructure posted Apr 20
- Member of Technical Staff — Inference-Kernel, Compiler & Communication posted Feb 17
- Member of Technical Staff — Inference-Multimodal & Diffusion posted Feb 17
- Member of Technical Staff — Inference-Core Engine posted Feb 17
- Member of Technical Staff — Cluster Infrastructure & Supercomputing posted Feb 17
- Member of Technical Staff — Inference-TPU posted Dec 30
From the team and dates on RadixArk's own job board.
Similar jobs
- Member of Technical Staff - Research Engineer, RL Environments at Preference Model
Shares 3 skills: CUDA, LLM, Machine Learning · Staff level · same country - Member of Technical Staff - Research Engineer at Black Forest Labs
Shares 2 skills: CUDA, LLM · Staff level · same country - Senior Machine Learning Engineer, Model Training and Reinforcement Learning at Nebius
Shares 3 skills: CUDA, LLM, Machine Learning · same city - AI Infrastructure Engineer at vCluster Labs
Shares 4 skills: CUDA, Node.js, LLM… · same country - Staff Engineer, Test Automation (R5792) at Shield AI
Shares 3 skills: CUDA, LLM, Machine Learning · Staff level · same country - Member of Technical Staff — Security at RadixArk
Also at this company
Is this your posting? Email info@careerholo.com to have it removed.