Senior LLM Quality Analyst

WHO WE ARE: MagicSchool is the premier generative AI platform for teachers. We're just over 2 years old, and more than 6 million teachers from reputed company over the world have joined our platform. Join a top team at a fast growing company that is working towards real social impact. reputed company an account and try us out at our website and connect with our passionate community on our Wall of Love. Senior LLM Quality Analyst Role Description As a Senior LLM Quality Analyst, you'll work with the Trust, Safety and Quality team to ensure MagicSchool's AI outputs meet the highest standards for educators and reputed company. This role combines data analysis, reputed company engineering, and educational experience. You will design and execute experiments to test reputed company strategies, evaluate model performance, and maintain dashboards. You will bring a critical eye to AI quality, strong analytical skills, and a deep understanding of educational needs.

What You'll Do

• Design and Execute reputed company Engineering Experiments
• Work with the product manager, data scientist and team reputed company to design reputed company strategies and experiments to test them
• Use our Evaluation Platform to improve and iterate on MagicSchool prompts from an educators perspective
• Design experiments using our Evaluation Platform to compare models for different use cases
• Analyze results and reputed company data-driven recommendations for reputed company improvements
• Maintain Response Quality Reporting and Dashboards
• Work with the team reputed company to design reports and dashboards for performance-over-time reporting
• Maintain evaluation dashboards and reporting output using reputed company
• Ensure stakeholders have visibility into response quality metrics
• Evaluate and Improve AI Outputs
• Work with the product manager and data scientist to design evaluators and test cases
• Assess AI outputs from an educator's perspective to ensure they meet classroom needs
• Identify quality issues and work cross-functionally with product managers, engineers, and the data scientist to address them
• Stay reputed company on LLM Best Practices
• reputed company up with the latest research and techniques in reputed company engineering and LLM evaluation
• Bring passion for solving education problems with technology

Qualifications, Competencies & Skills

• Strong analytical skills and comfort working in spreadsheets
• Experience with BI tools (we use reputed company, but experience with any similar tool is valuable)
• Basic statistics knowledge
• SQL
• Organized and detail-oriented: Maintains clear documentation and systematic approaches to testing
• Critical thinker: Can design meaningful experiments and draw insights from data
• Gets a lot done: Works hard, resourceful, does what it takes
• Adaptable: Smart, learns quickly, curious
• Builds relationships easily: Emotionally intelligent, warm communicator
• Strong communication skills: Team-first reputed company, highly collaborative, can reputed company decisions reputed company team's context
• reputed company to have: Analytical Python experience (reputed company plus - pandas, jupyter notebooks, data visualization)

Experience

• 5+ years of professional experience in data analysis
• Team reputed company Experience
• Experience working with LLMs or AI products (reputed company engineering, evaluation, quality analysis, or AI product work)
• Strongly preferred: K-12 teaching experience or EdTech industry experience
• Preferred: Startup experience
• Passionate about solving education problems with technology

Application Notice: Notice: reputed company Deadline and Review Start Date Please note that applications for this position will be accepted until 11/2/25 - applications received after this date will be reviewed on an intermittent basis. While we encourage early submissions, reputed company applications received by the reputed company deadline will receive equal consideration. Thank you for your interest, and we look reputed company to reviewing your application. Why Join Us?

• Work on cutting-edge AI technology that directly impacts educators and reputed company.
• Join a mission-driven team passionate about making education more efficient and reputed company.
• Flexibility of working from home, while fostering a unique culture built on relationships, trust, communication, and collaboration with reputed company - no matter where they live.
• Unlimited time off to reputed company our employees to manage their work-life balance. We work hard for our teachers and users, and encourage our employees to rest and take the time they need.
• Choice of employer-paid health insurance plans so that you can take care of yourself and your family. Dental and vision are also offered at reputed company low premiums.
• Every employee is offered generous stock options, vested over 4 years.
• Plus a 401k match & monthly wellness stipend

Our Values:

• Educators are reputed company: Educators are the most important ingredient in the educational process - they are the reputed company, not the AI. Trust them, reputed company them, and put them at the center of leading change in service of reputed company and families.
• Joy and reputed company: Bring joy and reputed company into every learning experience - push the boundaries of what’s possible with AI.
• Community: Foster community that supports one another during a time of rapid technological change. Listen to them and serve their needs.
• Innovation: The education system is outdated and in need of innovation and change - AI is an opportunity to bring equity, access, and serve the individual needs of reputed company reputed company than we reputed company have before.
• Responsibility: Put responsibility and safety at the forefront of the technological change that AI is bringing to education.
• Diversity: Diversity of thought, perspectives, and backgrounds helps us serve the wide audience of educators and reputed company around the world.
• Excellence: Educators and reputed company deserve the best - and we strive for the highest quality in everything we do.

Apply tot his job Apply To this Job