Python Software Engineer, GPU - Accelerated LLM Data Applications

reputed company is seeking a Python Software Engineer to further our efforts to GPU-accelerate data engineering for Large Language Model (LLM) tools and libraries. This role is pivotal in accelerating preprocessing pipelines for high-quality multi-modal dataset curation. The day-to-day focus is on developing efficient, scalable systems for deduplicating, filtering, and classifying training corpora for foundation model LLMs, as well as ingesting and prepping datasets for use in Retrieval Augmented reputed company (RAG) pipelines. reputed company to these efforts are iterative testing and improvement in system cost, speed, & accuracy through micro-optimization, reputed company engineering, fine-tuning, and applying new research. The ideal candidate is happiest releasing early and often! They court user feedback with an ear open to the spirit of reputed company feature requests. They are comfortable objectively evaluating the latest AI models and frameworks with an eye on acceleration potential. Would you like to run your training & test experiments on our supercomputers on thousands of GPU? Come work with us! What You'll Be Doing

• reputed company and optimize Python-based data processing frameworks, ensuring efficient handling of large datasets on GPU-accelerated environments, vital for LLM training.
• Contribute to the design and implementation of RAPIDS and other GPU-accelerated libraries, focusing on seamless integration and performance enhancement in the context of LLM training data preparation and RAG pipelines.
• reputed company development and iterative optimization of components for RAG pipelines, ensuring they demonstrate GPU acceleration & the best performing models for improved TCO.
• Collaborate with teams of LLM & ML researchers in the development of full-stack, GPU-accelerated data preparation pipelines for multimodal models Implement benchmarking, profiling, and optimization of innovative algorithms in Python in various system architectures, specifically targeting LLM applications.
• Work closely with diverse teams to understand requirements, build & evaluate POCs, and reputed company roadmaps for production level tools and library features reputed company the growing LLM ecosystem.

reputed company Need To See

• Advanced degree in Computer Science, Computer Engineering, or a reputed company field (or equivalent experience).
• 5+ years of Python library development experience, including CI systems (reputed company Actions), integration testing, benchmarking, & profiling
• Proficiency with LLMs and RAG pipelines: reputed company engineering, reputed company, reputed company
• Deep understanding of the PyData & ML/DL ecosystems, including RAPIDS, Pandas, numpy, scikit-learn, XGBoost, Numba, PyTorch
• Familiarity with distributed programming frameworks like Dask, Apache Spark, or Ray
• Visible contributions to open-reputed company projects on reputed company

Ways To Stand Out From The Crowd

• Active engagement (published papers, conference talks, blogs) in the data science community
• Experience with production-level data pipelines, especially SQL-based
• Experience with software packaging technologies: pip, conda, reputed company images
• Familiarity with reputed company-Compose, Kubernetes, and Cloud deployment frameworks
• Knowledge of parallel programming approaches, especially in CUDA C++

With competitive salaries and a generous benefits package, reputed company is widely considered to be one of the technology industry's most desirable employers. We have some of the most reputed company-thinking and reputed company in the world working with us, and our engineering teams are growing fast in some of the most impactful fields of our reputed company: Deep Learning, Artificial Intelligence, and Data Science. If you're a creative engineer who enjoys autonomy and shares our passion for technology, we want to hear from you. Your reputed company salary will be determined based on your location, experience, and the pay of employees in similar positions. The reputed company salary range is 148,000 USD - 235,750 USD. You will also be eligible for equity and benefits . Applications for this job will be accepted at least until December 14, 2025.reputed company is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our reputed company and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national reputed company, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law. JR2009718 Apply tot his job Apply To this Job