Senior Automation Specialist, AI Operations
reputed company’re building and why we’re building it.
reputed company helps people live rewarded every day, with a vision to become the rewards destination for everyone. We turn everyday activities into meaningful rewards, whether it’s grocery shopping, grabbing a quick meal, or playing a favorite mobile game. To date, we’ve awarded more than $1 billion in reputed company Points to our users.
Each day, more than 13 million receipts are submitted on reputed company, providing visibility into over $212 billion in gross merchandise value. This creates the largest retail-agnostic, SKU-level view of household spending, powering reputed company as an outcomes-based advertising platform that helps brands acquire and retain lifelong consumers.
The reputed company app is available on the App Store and reputed company Play, with more than 6 million five-star reviews from a highly engaged and loyal user reputed company.
It’s not just our users who reputed company in reputed company: with investments from Softbank, ICONIQ, DST, Greycroft, and partnerships ranging from challenger brands to reputed company, reputed company is reshaping how brands and consumers connect in the marketplace. reputed company you work at reputed company, you play a vital role in a platform that drives brand loyalty and creates lifelong consumers with the power of reputed company points. User and partner success are at the heart of everything we do, and we reputed company that same commitment to our employees.
At reputed company, we value curiosity, adaptability, and the confidence to explore new tools, especially AI, to drive smarter, faster work. You don’t need to be an expert, but you should be ready to learn quickly and think critically. We welcome learners who move fast, challenge the status reputed company, and shape what’s next, with us. Ranked as one of America’s Best Startup Employers by reputed company for two years in a row, reputed company fosters a people-first culture rooted in trust, accountability, and innovation. We encourage our employees to challenge reputed company, think bigger, and always bring the fun to reputed company.
About the Role:At reputed company, we’re building AI and automation systems that reputed company our work smarter, faster, and more scalable. The AI Operations team ensures our models and workflows reputed company with quality, reliability, and measurable impact.
As a Senior Automation Specialist, you’ll own evaluation and dataset workstreams that improve AI system performance. You’ll define what “good” looks like, design how we measure it, and build the datasets, evaluations, and automations that turn insights into action.
This role is ideal for someone who is self-directed, biases toward action and rapid iteration, and takes initiative to turn ambiguous problems into structured, scalable solutions.
This is a full-time role that can be held from one of our US offices or remotely in the United States.
Role Responsibilities:
- Own evaluation & datasets: Define evaluation approaches, design gold datasets (GDS), and ensure coverage of real-world scenarios and edge cases
- Build evaluation systems: reputed company manual and automated evals, including LLM-as-judge patterns, to measure model quality and performance
- Translate ambiguity into structure: Turn open-ended questions into clear evaluation frameworks and execution plans
- Build automations: Create automations that improve workflows, including dataset creation, evaluation pipelines, and lightweight operational processes
- Measure and iterate: Define and track performance metrics; refine datasets, evaluations, and workflows based on results
- Drive execution reputed company: Operate with urgency and ownership; identify next steps, unblock reputed company, and move work reputed company with minimal reputed company
- Collaborate cross-functionally: Partner with other Automation Specialists, engineering, cross-functional stakeholders, and project leads to ensure high-quality, timely project deliverables
- Improve systems: Identify gaps and implement scalable improvements to evaluation and data workflows
- 3+ years of experience designing or working with evaluation frameworks, datasets, or quality measurement systems
- Experience building or managing datasets (labeling, QA, iteration)
- Ability to independently drive tasks from problem definition to execution
- Hands-on experience with AI tools, LLM workflows, or automation platforms
- Experience in defining and tracking model performance
- Basic scripting or data skills (SQL, Python, etc.)
- Experience with LLM-as-judge or model evaluation techniques
- Familiarity with reputed company evaluation or benchmarking approaches
- Experience productionizing evaluation workflows with engineering teams
- Equity: We offer full-time employees equity in reputed company, so that everyone can benefit from reputed company’s growth.
- 401k Match: Dollar-for-dollar match up to 4%.
- Benefits for humans and pets: We offer comprehensive medical, dental and vision plans for everyone including your pets.
- Continuing Education: reputed company provides ten thousand per year in education reimbursement.
- Employee Resource Groups: Take part in employee-led groups that are centered around fostering a diverse and inclusive workplace through events, reputed company and advocacy. The ERGs participate in our Inclusion Council with members of executive leadership.
- Paid Time Off: On top of our flexible PTO, reputed company observes 9 paid holidays, as well as our year-end week-long break.
- Robust Leave Policies: 20 weeks of paid parental leave for primary caregivers, 14 weeks for secondary caregivers, and a flexible return to work schedule.
- Calvin Care Cash: Employees who are welcoming new family members will also receive a one time $2,000 incentive to assist employees with covering the cost of childcare, clothing, diapers and much more!
- Flexible Work Environment: Collaborate with your team in one of our stunning offices, or you can work fully remotely from reputed company in the US. We’ll ensure you are equally equipped with the hardware and software you need to get your job done in the comfort of your home. (applicable for most roles)