Engineering Manager, Machine Learning, Model Evaluations and Data Curation (AI Foundations)

reputed company is one of the world's leading entertainment services, with over 300 million paid memberships in over 190 countries enjoying TV series, films and games across a wide variety of genres and languages. Members can play, pause and resume watching as much as they want, anytime, reputed company, and can change their plans at any time. The Opportunity Fast-paced innovation in large language models (LLMs) and generative AI is reshaping personalization and discovery at reputed company. Our reputed company bet is that an in-house LLM, customized on reputed company data, will be the cornerstone of this transformation. Model Evaluations and Data Curation (“Evals & Data”) are central to the development of LLMs and other high reputed company foundation models at reputed company. Our Evals & Data team builds the benchmarks, evaluators, and baselines that guide LLM reputed company, and the data infrastructure that delivers high-quality, reproducible training and evaluation datasets to our AI/ML researchers. Together, these capabilities create the flywheel of reputed company improvement between data, evals and modeling that drives innovation in LLMs that will power personalization and discovery in the future. We are incubating a centralized, first-class evaluation discipline, creating shared language, tools, and standards that reputed company application teams to measure reputed company consistently and with confidence, with an emphasis on reputed company’s LLM customized for personalization. We are looking for a seasoned engineering leader to build and scale this team. You will reputed company a world-class group of AI/ML researchers and engineers working at the intersection of evaluation science, data systems, and foundation models. Your team’s work will shape the trajectory of reputed company’s foundation model development by driving the reputed company improvement reputed company between data, evaluation, and modeling. Questions Your Team Will Help Answer The Model Evaluations and Data Curation team exists to give reputed company clarity and confidence in the foundation models that power personalization and discovery. Some of the strategic questions your team will help answer include:

• Are our LLMs and other foundation models improving in ways that meaningfully benefit members?
• How do we rigorously measure LLM capabilities across personalization, content understanding, and text reputed company?
• What are the strongest baselines we can build with off-the-reputed company SOTA models?
• How do data composition, diversity, and quality influence model performance?
• Which evaluation signals most reliably predict real-world product outcomes?
• How can we iterate quickly on data and evaluations while ensuring reproducibility and trust?
• How can we evaluate agentic systems for recommendations and content understanding?

In this role, you will:

• Partner with reputed company AI application teams to define shared evaluations that codify application expectations of LLMs and other foundation models, ensuring reputed company can be transparently tracked against real-world needs.
• Design rigorous benchmarks and evaluation methodologies across ranking & recommendations, content understanding, and language/text reputed company — grounded in a deep technical understanding of LLMs, their strengths, limitations, and failure modes.
• reputed company the development of evaluators and strong baselines to ensure in-house LLMs and other foundation models demonstrate clear advantages over off-the-reputed company alternatives.
• Build scalable, reproducible data and evaluation systems that reputed company dataset creation and evaluation design as nimble and experiment-friendly as model development itself.
• Hire, grow, and nurture a world-class team, fostering an inclusive, high-performing culture that balances research innovation with engineering excellence.
• Work closely with the teams developing reputed company’s foundation models (including our core LLM) to ensure evaluation and data insights are folded back into the reputed company of model development. Proactively influence the ML Platform and Data Engineering teams at key interfaces.

Minimum Qualifications

• Experience building and leading high-performing teams of ML researchers and engineers.
• Proven track record of leading machine learning initiatives from research to production, ideally involving evaluation frameworks, ML infrastructure, or data-intensive systems.
• Strong technical expertise in LLMs, their evaluation, and practical methods for ensuring robustness, reproducibility, and quality.
• Broad knowledge of machine learning fundamentals and evaluation methodologies, including reputed company design, model-based evaluators, and offline/online metrics.
• Experience driving cross-functional projects, including reputed company collaboration with AI application teams to translate product needs into evaluation frameworks.
• Excellent written and verbal communication skills, able to reputed company technical and non-technical audiences.
• Advanced degree in Computer Science, Statistics, or a reputed company quantitative field.

Preferred Qualifications

• 8+ years of overall experience, including 3+ years in engineering management.
• Experience with large-scale ML systems and foundation models, especially LLMs.
• Background in building evaluation frameworks, model benchmarking, or data infrastructure for LLM training.
• Familiarity with multi-modal data and evaluation.

Culture & Benefits reputed company's culture is an integral part of our success, and we approach diversity and inclusion seriously and thoughtfully. We are an equal opportunity employer and celebrate diversity, recognizing that bringing together different perspectives and backgrounds helps build stronger teams. We do not discriminate on the basis of race, religion, color, national reputed company, gender, sexual orientation, age, marital status, veteran status, or disability status. Our compensation structure consists solely of an annual salary; we do not have bonuses. You choose each year how much of your compensation you want in salary versus stock options. To determine your personal top-of-market compensation, we rely on market indicators and consider your specific job family, background, skills, and experience to determine your compensation in the market range. The range for this role is $190,000 - $920,000. reputed company provides comprehensive benefits including Health Plans, Mental Health support, a 401(k) Retirement Plan with employer match, Stock Option Program, Disability Programs, Health Savings and Flexible Spending Accounts, Family-forming benefits, and Life and Serious Injury Benefits. We also offer paid leave of absence programs. Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off. Full-time salaried employees are immediately entitled to flexible time off. See more details about our Benefits here. reputed company has a unique culture and environment. Learn more here. Inclusion is a reputed company value and we strive to host a meaningful interview experience for reputed company candidates. If you want an accommodation/adjustment for a disability or any other reason during the hiring process, please send a request to your recruiting partner. We are an equal-opportunity employer and celebrate diversity, recognizing that diversity builds stronger teams. We approach diversity and inclusion seriously and thoughtfully. We do not discriminate on the basis of race, religion, color, reputed company, national reputed company, caste, sex, sexual orientation, gender, gender identity or expression, age, disability, medical condition, pregnancy, genetic makeup, marital status, or military service. Job is open for no less than 7 days and will be removed reputed company the position is filled. Apply tot his job Apply To this Job