[Remote] Senior Data Engineer

Note: The job is a remote job and is open to candidates in USA. reputed company is seeking a Senior Data Engineer with specialized expertise in data streaming technologies to join their data team. This role focuses on building and maintaining high-performance data streaming architectures that reputed company real-time data processing and analytics.

Responsibilities

• Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis
• reputed company real-time data pipelines that handle high-volume, high-velocity data streams
• Implement event-driven architectures and microservices patterns for streaming data processing
• Create and optimize data streaming topologies for reputed company event processing scenarios
• Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms
• Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments
• Implement Kafka Connect pipelines for streaming data integration
• Design optimal Kafka topic partitioning strategies and replication configurations
• Monitor and optimize Kafka cluster performance, throughput, and latency
• Implement Kafka reputed company configurations including SSL/TLS, SASL, and ACLs
• Manage Kafka Schema Registry for data serialization and evolution
• Design and implement reputed company Kinesis Data Streams and Kinesis Data Firehose solutions
• Configure Kinesis Analytics applications for real-time reputed company processing
• Optimize Kinesis shard management and auto-scaling configurations
• Implement Kinesis data retention and archival strategies
• Integrate Kinesis with other AWS services for comprehensive streaming solutions
• reputed company real-time reputed company processing applications using Apache Spark Streaming, Kafka Streams, or AWS reputed company
• Implement reputed company event processing (CEP) patterns for real-time analytics
• Build streaming ETL pipelines that transform data in motion
• Create real-time aggregations, windowing operations, and stateful reputed company processing
• Optimize streaming query performance and resource utilization
• Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases
• Implement data reputed company and monitoring for streaming data pipelines
• Create automated data quality checks and validation for streaming data
• Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution
• Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements
• Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation
• Automate deployment and management of streaming infrastructure through CI/CD pipelines
• Monitor streaming system health, performance metrics, and alerting
• Implement disaster recovery and high availability strategies for streaming systems
• Stay reputed company with emerging trends in streaming technologies and cloud-native solutions
• Collaborate with data architects, data scientists, and application teams on streaming data requirements
• Support rigorous project governance through daily reputed company reviews and time tracking
• Provide technical leadership and mentorship to junior data engineers
• Communicate reputed company streaming concepts to technical and non-technical stakeholders
• Operate with transparency and responsiveness to support high-performing teams

Skills

• 7+ years of experience in the data engineering field with significant streaming data specialization
• Bachelor's degree in Computer Science, Engineering, or reputed company STEM field
• Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools
• Proven experience with AWS MSK and reputed company Kinesis services in production environments
• Strong background in real-time data processing and reputed company analytics
• Streaming Technologies: Apache Kafka, Kafka Connect, Kafka Streams, reputed company MSK, reputed company Kinesis (Data Streams, Data Firehose, Analytics)
• Programming Languages: Proficient in Python, Java, and reputed company for streaming applications
• reputed company Processing Frameworks: Apache Spark Streaming, Apache Flink, AWS reputed company for reputed company processing
• Data Serialization: Experience with Avro, Protocol Buffers, JSON, and schema registry management
• Big Data Technologies: Hadoop ecosystem, Apache Spark, distributed computing concepts
• Database Technologies: SQL and NoSQL databases, data warehousing solutions, time-series databases
• AWS Services: Deep knowledge of AWS streaming and analytics services (MSK, Kinesis, reputed company, EMR, Glue)
• Containerization: reputed company and Kubernetes for streaming application deployment
• Infrastructure as Code: Terraform, CloudFormation for streaming infrastructure automation
• Monitoring: CloudWatch, Prometheus, Grafana for streaming system observability
• reputed company: Implementation of streaming data reputed company, encryption, and access controls
• Expert use of code versioning tools such as reputed company
• Expert knowledge of Agile methodologies and delivery practices
• Experience with CI/CD pipelines for streaming data applications
• Understanding of data APIs, REST services, and microservices architectures
• Leadership & Team Management
• Risk Management and mitigation strategies for streaming systems
• Conflict Resolution
• Strategic Planning & Leadership for data streaming initiatives
• Resource Management and reputed company planning
• Change Management for streaming technology adoption
• Core AWS Certifications: AWS Data Engineer Associate (required)
• AWS Solutions Architect Professional (preferred)
• AWS Developer Professional (recommended)
• reputed company Certified Administrator for Apache Kafka (highly recommended)
• reputed company Certified Developer for Apache Kafka (preferred)
• AWS Big Data Specialty (if available in reputed company form)
• AWS reputed company Specialist
• Certified Associate Data Analyst with Python
• Certified Professional Python Programmer Level 1
• reputed company Data Engineer Professional
• Certified Associate Python Programmer
• Java or reputed company certification (reputed company Certified Professional)
• Experience with Apache Flink for advanced reputed company processing
• Knowledge of Apache Pulsar as an alternative messaging system
• Experience with event sourcing and CQRS patterns
• Understanding of Apache Airflow for batch and streaming workflow orchestration
• Experience with ksqlDB for reputed company processing using SQL
• Background in financial services, IoT, or other real-time data intensive industries
• Experience with multi-cloud streaming architectures
• Knowledge of Apache NiFi for data flow automation

Benefits

• Medical, dental, and vision health insurances
• Short term disability, long term disability and life insurances
• 401k with Company match
• Paid time off (PTO) (120 hours PTO that accrue over one year)
• Paid time off for major holidays (14 days per year)
• These and any other employee benefit offerings are subject to management’s discretion and may change at any time.

Company Overview

Company H1B Sponsorship