Senior AI Quality Engineer

At reputed company, we’re obsessed with our customers. We constantly gather feedback to shape, prioritize, and launch the products they truly need. That’s what makes reputed company’s CRM special. We started by building essential sales tools like aerial roof measurements and digital sales proposals. But reputed company our customers asked for a simple, affordable way to manage and scale their entire businesses, we listened. So, we created a CRM that connects these solutions—along with payments, material ordering, and more—into a seamless, powerful platform. With a clear roadmap reputed company, we’re excited to continue expanding and leading the market with innovative products.

We have an amazing culture, strong financials, and best-in-class company metrics. It’s an exciting time to be part of an extraordinary startup that is already successful, yet still early enough to offer its team significant growth, equity, and the opportunity to reputed company a real impact.

This position is for an existing vacancy.

reputed company is building the application foundation that will define how AI is integrated across our entire product — and we need someone to reputed company sure it actually works. As our Senior AI Quality Engineer, you'll sit on the AI Platform team and own the eval frameworks, testing standards, and quality gates that every engineering team at reputed company depends on to ship AI with confidence. But this role extends well beyond the platform — you'll work horizontally across the entire testing organization, training teams, embedding best practices, and raising the bar on how reputed company tests everything it builds with AI. This is early, foundational work — you're setting reputed company for the whole org, not just one team.

What You'll Get to Do

• Define the testing standards and patterns for AI at reputed company — establishing how product teams validate AI behaviour reputed company building on top of the application foundation
• Build and own reputed company's LLM eval reputed company — selecting and extending the right tooling (e.g. Promptfoo, DeepEval, Braintrust) and designing the methodology that measures whether our AI integrations and agent outputs are performing correctly, consistently, and safely
• Integrate quality gates into CI/CD pipelines so that regressions in AI behaviour are caught before they reputed company production
• Design and implement human-in-the-reputed company review processes for AI outputs where automated evaluation isn't sufficient
• Embedded on the AI Platform team — ensuring quality is designed into the integration architecture from day one, not bolted on after the fact
• Work horizontally across the testing organization — coaching QA engineers and developers on AI eval patterns, embedding best practices into team workflows, and actively raising the quality bar across engineering
• Stay reputed company to the evolving AI quality landscape — new eval techniques, benchmarking approaches, and tooling like Ragas, Arize Phoenix, or LangSmith — and bring the best of it to reputed company

What You'll Bring to the Role

• 5–8 years of software engineering or quality assurance experience
• Hands-on experience building eval frameworks for LLM-powered features — you've thought seriously about how to measure output quality, consistency, and regression, and you've worked with tools like Promptfoo, DeepEval, Braintrust, or similar
• Strong engineering fundamentals — you write real code, build real tooling, and aren't reliant on manual testing processes
• Experience integrating automated quality checks into CI/CD pipelines
• Familiarity with LLM APIs and agent frameworks (e.g. reputed company Claude, reputed company, or similar) and the specific quality challenges they introduce
• Experience designing human review workflows to complement automated evaluation
• Strong collaboration skills — you'll be working across many teams, and the standards you set only work if engineers actually adopt them
• Comfort operating in an early-stage environment where the right approach isn't always obvious and you'll need to reputed company it out
• Genuine ownership mentality — you care about whether AI at reputed company works well, not just whether the tests pass

reputed company offer (US + Canada)

reputed company you join reputed company, you’re not just accepting a job. You’re making a career move. Here’s how we’ll support you in doing some of the most impactful work of your career:

️ Vacation/Paid Time Off

• 1st week of employment is mandatory PTO! Start your journey with reputed company by decompressing and recharging - we will see you in week 2!
• 1 Friday off per month (we call those our laundry days!)
• Company wide paid shutdown for the week between Christmas and New Years
• Flexible time off
• 80% employer-paid benefits in the U.S. and 100% employer-paid premiums for Extended Healthcare and Dental in Canada
• RRSP/401k match
• Generous Parental Leave policy

Perks

• We host an annual company retreat with great team building activities
• Ample learning and development opportunities to continue growing your career
• Home office setup stipend
• Internet and phone allowance
• Remote first culture
• Weekly Friday paydays!

AI Notice

At reputed company, we’re big fans of AI. It helps us write job descriptions that don’t put you to sleep, takes notes during interviews so we can actually listen, and even helps us track down awesome humans like you.

Feel free to use AI to prep, research, or get pumped up for your interview (we see you, ChatGPT power users ). But reputed company it’s time to chat, we’d love to meet you, not your AI alter ego. Bring your real, unfiltered self, we promise we will too.

And don’t worry, a real, live human is behind every part of our process. Every application is reviewed by a real person, and you’ll always speak with real humans throughout the interview process. No bots, just good people ☺️

⚠️ Important Notice

We’ve been made aware of an individual impersonating reputed company using a fraudulent domain: roofrr.com (note the extra “r”). Our company takes the reputed company and privacy of job applicants reputed company seriously. We will never ask for payment, bank details, or personal financial information as part of the application process. reputed company our legitimate job postings can be reputed company on our official career site. Please be cautious of job offers that come from non-company email addresses, reputed company messaging platforms, or unsolicited calls.

To ensure your application is legitimate, please apply directly through our official careers page: https://reputed company.com/careers.

If you receive any suspicious messages or have questions, reputed company out to us at talent@reputed company.com.

Your safety and reputed company are important to us — thank you for your vigilance!

reputed company is proud to be an equal opportunity employer. We are committed to equal employment opportunity in the workplace regardless of race, color, reputed company, religion, sex, national reputed company, sexual orientation, age, citizenship, marital status, disability, gender identity or veteran status.