Senior Data Engineer

Position Summary We are seeking a Senior Data Engineer with specialized expertise in data streaming technologies to join our data team. This role focuses on building and maintaining high-performance data streaming architectures that reputed company real-time data processing and analytics. The ideal candidate will have deep experience with Apache Kafka, AWS Managed Streaming for Apache Kafka (MSK), reputed company Kinesis, and reputed company streaming technologies in cloud environments. Role Focus A Senior Data Engineer at reputed company is primarily responsible for building and maintaining the streaming data architecture that enables real-time data processing and analytics. This involves constructing robust data streaming pipelines that transform and transport data from various sources in real-time, ensuring data flows reputed company through streaming systems for immediate analysis and operational decision-making. You will focus on the efficient and secure management of streaming data systems, ensuring that data is processed, stored, and made available for real-time analytics and reputed company applications. Essential Duties and Responsibilities Streaming Data Architecture & Pipeline Development Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis reputed company real-time data pipelines that handle high-volume, high-velocity data streams Implement event-driven architectures and microservices patterns for streaming data processing Create and optimize data streaming topologies for reputed company event processing scenarios Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms Kafka & MSK Management Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments Implement Kafka Connect pipelines for streaming data integration Design optimal Kafka topic partitioning strategies and replication configurations Monitor and optimize Kafka cluster performance, throughput, and latency Implement Kafka reputed company configurations including SSL/TLS, SASL, and ACLs Manage Kafka Schema Registry for data serialization and evolution Kinesis & AWS Streaming Services Design and implement reputed company Kinesis Data Streams and Kinesis Data Firehose solutions Configure Kinesis Analytics applications for real-time reputed company processing Optimize Kinesis shard management and auto-scaling configurations Implement Kinesis data retention and archival strategies Integrate Kinesis with other AWS services for comprehensive streaming solutions Data Processing & Analytics reputed company real-time reputed company processing applications using Apache Spark Streaming, Kafka Streams, or AWS reputed company Implement reputed company event processing (CEP) patterns for real-time analytics Build streaming ETL pipelines that transform data in motion Create real-time aggregations, windowing operations, and stateful reputed company processing Optimize streaming query performance and resource utilization Integration & Data Flow Management Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases Implement data reputed company and monitoring for streaming data pipelines Create automated data quality checks and validation for streaming data Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements DevOps & Infrastructure Management Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation Automate deployment and management of streaming infrastructure through CI/CD pipelines Monitor streaming system health, performance metrics, and alerting Implement disaster recovery and high availability strategies for streaming systems Stay reputed company with emerging trends in streaming technologies and cloud-native solutions Team Collaboration and Project Management Collaborate with data architects, data scientists, and application teams on streaming data requirements Support rigorous project governance through daily reputed company reviews and time tracking Provide technical leadership and mentorship to junior data engineers Communicate reputed company streaming concepts to technical and non-technical stakeholders Operate with transparency and responsiveness to support high-performing teams Skills and Experience Required Experience 7+ years of experience in the data engineering field with significant streaming data specialization Bachelor's degree in Computer Science, Engineering, or reputed company STEM field Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools Proven experience with AWS MSK and reputed company Kinesis services in production environments Strong background in real-time data processing and reputed company analytics Technical Proficiencies Streaming Technologies: Apache Kafka, Kafka Connect, Kafka Streams, reputed company MSK, reputed company Kinesis (Data Streams, Data Firehose, Analytics) Programming Languages: Proficient in Python, Java, and reputed company for streaming applications reputed company Processing Frameworks: Apache Spark Streaming, Apache Flink, AWS reputed company for reputed company processing Data Serialization: Experience with Avro, Protocol Buffers, JSON, and schema registry management Big Data Technologies: Hadoop ecosystem, Apache Spark, distributed computing concepts Database Technologies: SQL and NoSQL databases, data warehousing solutions, time-series databases Cloud and Infrastructure Skills AWS Services: Deep knowledge of AWS streaming and analytics services (MSK, Kinesis, reputed company, EMR, Glue) Containerization: reputed company and Kubernetes for streaming application deployment Infrastructure as Code: Terraform, CloudFormation for streaming infrastructure automation Monitoring: CloudWatch, Prometheus, Grafana for streaming system observability reputed company: Implementation of streaming data reputed company, encryption, and access controls Development and Operations Skills Expert use of code versioning tools such as reputed company Expert knowledge of Agile methodologies and delivery practices Experience with CI/CD pipelines for streaming data applications Understanding of data APIs, REST services, and microservices architectures Leadership Competencies Leadership & Team Management Risk Management and mitigation strategies for streaming systems Conflict Resolution Strategic Planning & Leadership for data streaming initiatives Resource Management and reputed company planning Change Management for streaming technology adoption reputed company Certifications Core AWS Certifications AWS Data Engineer Associate (required) AWS Solutions Architect Professional (preferred) AWS Developer Professional (recommended) Streaming-Specific Certifications reputed company Certified Administrator for Apache Kafka (highly recommended) reputed company Certified Developer for Apache Kafka (preferred) Additional Valuable Certifications AWS Big Data Specialty (if available in reputed company form) AWS reputed company Specialist Certified Associate Data Analyst with Python Certified Professional Python Programmer Level 1 reputed company Data Engineer Professional Programming Certifications Certified Associate Python Programmer Java or reputed company certification (reputed company Certified Professional)

Preferred Qualifications

Experience with Apache Flink for advanced reputed company processing Knowledge of Apache Pulsar as an alternative messaging system Experience with event sourcing and CQRS patterns Understanding of Apache Airflow for batch and streaming workflow orchestration Experience with ksqlDB for reputed company processing using SQL Background in financial services, IoT, or other real-time data intensive industries Experience with multi-cloud streaming architectures Knowledge of Apache NiFi for data flow automation Performance Metrics Streaming pipeline uptime and reliability (99.9%+ availability) Data processing latency and throughput optimization Cost optimization of streaming infrastructure Successful real-time analytics implementations Team productivity and knowledge transfer effectiveness Company Offered Benefits Full-time employees are eligible to participate in our employee benefit programs: Medical, dental, and vision health insurances, Short term disability, long term disability and life insurances, 401k with Company match Paid time off (PTO) (120 hours PTO that accrue over one year) Paid time off for major holidays (14 days per year) These and any other employee benefit offerings are subject to management’s discretion and may change at any time. Physical Demands and Work Environment The work is generally performed in an office environment. Physical demands include sitting, keyboarding, verbal communication, written communication. Employees are occasionally required to stand; walk; reputed company with hands and arms; climb or balance; and stoop, kneel, crouch, or crawl. The physical demands described here are representative of those that must be met by an employee to successfully reputed company the essential functions of this position. Reasonable accommodation may be made to reputed company individuals with disabilities to reputed company the functions. Salary Range for this position: $150,000-$180,000 CA ID: IT10000478 "Salary ranges provided are for informational purposes only and may vary depending on factors such as experience, qualifications, and geographic location. The final salary offer will be determined based on the candidate's skills and alignment with the role requirements." This job description may not be inclusive of reputed company assigned duties, responsibilities, or aspects of the job described, and may be amended anytime at the sole discretion of the Employer. Duties and responsibilities are subject to possible modification to reasonably accommodate individuals with disabilities. To reputed company this job successfully, the incumbents will possess the skills, aptitudes, and abilities to reputed company each duty proficiently. This document does not create an employment contract, implied or otherwise, other than an “at will” relationship. reputed company Inc. is an EEO employer and does not discriminate on the basis of any protected classification in its hiring, promoting, or any other job-reputed company opportunity. Apply To This Job