Director of Machine Learning Engineering

About Us

Chalice Custom Algorithms (chalice.ai) is the leading AI solution for brands applying their own data and analytics to real-time decisioning in ad buying. Our platform automates data ingestion, predictive analytics, and the deployment of custom bidding logic across all major DSPs, Meta, and YouTube. Chalice has been recognized as “Best Demand Side Tech” by AdExchanger and powered AdWeek’s “Best Use of Programmatic” in the 2023 Media Plan of the Year awards.

About the Role: We are seeking a Director of Machine Learning Engineering (MLE) to lead the architecture, deployment, and optimization of our machine learning infrastructure and models. This role will report directly to the VP of Data Science and Analytics, manage our Senior MLE(s), and collaborate closely with the Engineering Architect to design and scale robust, low-latency enterprise pipelines. The Director will also mentor junior software engineers in machine learning engineering best practices, fostering growth across the team.

This is a high-impact position for a leader who thrives at the intersection of ML infrastructure, production systems, and business value delivery. You will help define the technical vision, ensure delivery of scalable AI solutions, and drive cross-functional initiatives that shape our platform’s evolution.

Key Responsibilities

Lead the design, development, and deployment of scalable machine learning systems leveraging Databricks, PySpark, Rust, and Kubernetes for both low-latency applications and offline training pipelines.

• Oversee LLMs and neural network predictive modeling pipelines, ensuring performance, observability, and alignment with product strategy.

• Own the full ML lifecycle—from data ingestion and feature engineering to model deployment and performance monitoring.

• Collaborate with the Engineering Architect to build robust, fault-tolerant, and maintainable ML pipelines that support enterprise scale.

• Manage and mentor a growing team of 3 senior and junior MLEs and collaborate closely with software engineers, fostering technical excellence, code quality, and architectural rigor.

Own the end-to-end machine learning lifecycle from data ingestion and feature engineering through model deployment and monitoring.

• Partner with product, data science, and engineering teams to translate business objectives into technical solutions that drive measurable outcomes.

• Guide adoption of emerging technologies and techniques (e.g., LoRA, quantization, vector stores, Rust microservices for model serving) to improve model efficiency and latency.

• Ensure compliance with audit, privacy, and security standards in all ML engineering efforts.

Qualifications

• 8+ years of industry experience in machine learning engineering, with at least 3+ years leading ML engineering teams.

• Deep experience building low-latency machine learning systems using Kubernetes, Rust, PySpark, and Databricks, Pytorch, Tensorflow, and state of the art neural network packages.

• Proven track record in designing and deploying LLM- and neural network-based solutions at scale.

• Experience with CI/CD Integrations and ML Data and Model versioning

• Mastery of MLOps tools and processes (MLflow, Prometheus, Grafana, Airflow, etc.).

• Strong experience with cloud platforms (AWS, Databricks) and production data engineering at scale.

• Ability to influence architecture and strategy while being hands-on when necessary.

• Experience in mentoring and developing technical talent across seniority levels.

• Excellent communication and collaboration skills to partner across product, engineering, and executive teams.

• Advertising technology or performance marketing experience is a strong plus.

Why Join Us

At Chalice, you’ll be at the forefront of AI-driven advertising innovation. We offer:

• A collaborative, high-performance environment where your work shapes the future of custom AI for brands.

• Competitive compensation plus performance incentives.

• Comprehensive benefits (medical, dental/vision, 401k options, unlimited PTO, 11 company holidays, year-end closure).

• A culture that values transparency, non-conformity, and diversity of thought.