Khan Academy is hiring a Senior Platform Engineer I, AI Evaluation (24 months fixed-term)

Responsibilities

• Be fluent in the range of offline and online evaluation strategies, and when to apply the techniques over the lifecycle of development
• Have intuitions about how to specify eval pipelines succinctly using declarative syntax
• Understand the role of stratified datasets and ground truth labeling
• Appreciate the range of eval scoring schemes from human raters to automated LLMs-as-judge

Requirements

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, related field, or equivalent professional experience.
• 5 years of Software Engineering including significant time working on the evaluation of generative AI systems or other evaluations of ML model quality
• Strong programming skills in Go, Python, SQL, and at least one data pipeline framework (e.g., Airflow, Dagster, Prefect)
• Familiarity with the architecture of large language models and their industry-standard APIs

Nice to Have

• Experience with labeling platforms (e.g., Label Studio, Scale AI, Toloka) and human-in-the-loop concerns such as rubric development and inter-rater agreement
• Exposure to MLOps practices such as model registry, feature store, or continuous evaluation
• Background in education technology or other human-centered AI applications

Benefits

• Competitive salaries
• Ample paid time off as needed
• 8 pre-scheduled Wellness Days in 2026
• Remote-first culture
• Generous parental leave
• An exceptional team that trusts you and gives you the freedom to do your best
• The chance to put your talents towards a deeply meaningful mission and the opportunity to work on high-impact products that are already defining the future of education
• Opportunities to connect through affinity, ally, and social groups
• 401(k) + 4% matching & comprehensive insurance, including medical, dental, vision, and life

Work Arrangement

Hybrid

Additional Information

• As part of our hiring process, we use a secure identity verification service through CLEAR® (in partnership with Greenhouse) to confirm that each applicant is who they claim to be. CLEAR® provides a safe, consistent way to confirm identity, helping protect both applicants and the company from impersonation or fraud.
• 24 months fixed-term