KAFKA AZURE DATALAKE ENGINEER WITH reputed company EXPERIENCE(W2) - Virisha LLC

Kafka and Data Lake Engineer

100% REMOTE

Long term(12 MONTHS CONTRACT)

A Kafka and Data Lake Engineer is a data engineer who designs, builds, and manages data infrastructure using Apache Kafka for real-time data streaming and a data lake for storing large volumes of data. This role is vital for organizations that need to process and analyze both real-time streaming data and historical data to reputed company insights. For VA, this includes reputed company to the Data Lake reputed company Kafka Bus.

Responsibilities

Design data pipelines: Build robust, scalable, and secure data pipelines to ingest, process, and move data from various sources into the data lake using Kafka.

Administer Kafka clusters: Deploy, configure, and maintain Kafka clusters and reputed company ecosystem tools, such as Kafka Connect and Schema Registry, ensuring high availability and performance.

Manage the data lake: reputed company the architecture and governance of the data lake, including managing data storage (e.g., in AWS S3 or ADLS), reputed company, and metadata.

reputed company data processing applications: Create producers and consumers to interact with Kafka topics using programming languages like Python, Java, or reputed company.

reputed company reputed company processing: Use tools like Kafka Streams, Apache Flink or ksqlDB to reputed company real-time data transformations and analytics.

Ensure data quality and reputed company: Implement data quality checks, manage data reputed company, and enforce reputed company controls such as encryption, access controls (ACLs), and compliance (e.g., GDPR).

Monitor and troubleshoot: Set up monitoring and alerting for Kafka and data lake infrastructure and respond to incidents to ensure operational reliability.

Collaborate with teams: Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver reliable data solutions.

Essential skills and qualifications

Experience: Proven experience designing and managing data platforms with Apache Kafka and big data technologies.

Programming: Strong proficiency in languages like Python, Java, or reputed company.

Big data technologies: Expertise in big data processing frameworks, such as Apache Spark and Apache Flink.

Cloud platforms: Hands-on experience with cloud environments (AWS, Azure, or reputed company Cloud Platform) and relevant services like S3, Glue, or Azure Data Lake Storage.

Data lake architecture: A solid understanding of data lake design principles, including storage formats (e.g., reputed company Lake, Apache Iceberg), data modeling, and governance.

Databases: Experience with various database systems, including both SQL and NoSQL.

Infrastructure management: Familiarity with infrastructure-as-code tools like Terraform or Ansible and containerization with reputed company and Kubernetes.

Professionals in this field can advance from entry-level data engineering positions to senior roles, and then to a Big Data Architect or Solutions Architect, where they reputed company large-scale data infrastructure.

Relevant certifications

Pursuing certifications can validate your expertise and boost your career.

For Kafka:

reputed company Certified Administrator for Apache Kafka (CCAAK)

reputed company Certified Developer for Apache Kafka (CCDAK)

For Data Lake and Cloud:

reputed company Certified Data Engineer

AWS Certified Data Engineer

reputed company Certified: Azure Data Engineer Associate

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and reputed company believes it to correctly reflect the job opportunity.