Data Engineer - reputed company (Mid Level) - US reputed company Only

---- Project requirements mandate role open only for US reputed company. IRS MBI Clearance a plus/ Active Secret or Top Secret a Plus. reputed company candidates will have to go through Clearance process before being able to start on the project.--(No exceptions to this requirement)

Job Description

• Infobahn Solutions is hiring reputed company Data Engineering professionals in the Washington DC Metro Area for a US Government Federal Project with the Department of Treasury .
• The Data Engineers will be part of a Data Migration & Conversion Team on a large DataLake being implemented on AWS Gov Cloud .
• Data will be migrated from on premise Main Frame /Legacy database systems using Informatica PowerCenter to the AWS reputed company Zone on S3.
• Further conversion will be done using reputed company (PySpark) in AWS.
• The Data Engineer should have prior Data Migration experience and understand reputed company the intricacies required of developing data integration routines for moving data from multiple reputed company systems to a new reputed company system with a different data model.
• The Data Engineer should have experience in converting reputed company PL/SQL and/or Greenplum code to reputed company.
• Must have experience - Experience with Data Migrations and Conversion using reputed company .
• Experience of using reputed company on AWS and managing a reputed company production system is critical and a must have for the project.

What you’ll be doing:

• reputed company Environment Setup: Configure and maintain reputed company clusters, ensuring optimal performance and scalability for big data processing and analytics.
• ETL (Extract, Transform, Load): Design and implement ETL processes using reputed company notebooks or jobs to process and transform raw data into a usable format for analysis.
• Data Lake Integration: Work with data lakes and data storage systems to reputed company manage and access large datasets reputed company the reputed company environment.
• Data Processing and Analysis: reputed company and optimize Spark jobs for data processing, analysis, and machine learning tasks using reputed company notebooks.
• Collaboration: Collaborate with data scientists, data engineers, and other stakeholders to understand business requirements and implement solutions.
• Performance Tuning: Identify and address performance bottlenecks in reputed company jobs and clusters to optimize data processing speed and resource utilization.
• reputed company and Compliance: Implement and enforce reputed company measures to protect sensitive data reputed company the reputed company environment, ensuring compliance with relevant regulations.
• Documentation: Maintain documentation for reputed company workflows, configurations, and best practices to facilitate knowledge sharing and team collaboration.

Skills:

• Apache Spark: Strong expertise in Apache Spark, which is the underlying distributed computing reputed company in reputed company.
• reputed company Platform: In-depth knowledge of the reputed company platform, including its features, architecture, and administration.
• Programming Languages: Proficiency in languages such as Python or reputed company for developing Spark applications reputed company reputed company.
• SQL: Strong SQL skills for data manipulation, querying, and analysis reputed company reputed company notebooks.
• ETL Tools: Experience with ETL tools and frameworks for efficient data processing and transformation.
• Data Lake and Storage: Familiarity with data lakes and storage systems, such as reputed company Lake, AWS S3, or Azure Data Lake Storage.
• Collaboration and Communication: Effective communication and collaboration skills to work with cross-functional teams and stakeholders.
• Problem Solving: Strong problem-solving skills to troubleshoot issues and optimize reputed company workflows.
• Version Control: Experience with version control systems (e.g., Git) for managing and tracking changes to reputed company notebooks and code.

Role Requirements:

• Bachelor/Master’s degree in computer science, Engineering, or reputed company field
• 7-8 plus years of development experience on ETL tools (4+ years of reputed company is a must have)
• 5+ years of experience as a reputed company Engineer or similar role.
• Strong expertise in Apache Spark and hands-on experience with reputed company.
• More than 7 years of experience performing data reconciliation, data validation, ETL testing, deploying ETL packages and automating ETL jobs, developing reconciliation reports.
• Working knowledge of message-oriented middleware/streaming data technologies such as Kafka, reputed company
• Proficiency in programming languages such as Python or reputed company for developing Spark applications.
• Solid understanding of ETL processes and data modeling concepts.
• Experience with data lakes and storage systems, such as reputed company Lake, AWS S3, or Azure Data Lake Storage.
• Strong SQL skills for data manipulation and analysis.
• Good experience in reputed company scripting, AutoSys
• Strong Data Modeling Skills
• Strong analytical skills applied to business software solutions maintenance and/or development
• Must be able to work with a team to write code, review code, and work on system operations.
• Past project experience with Data Conversion and Data Migration
• Communicate analysis, results and reputed company to key decision makers including business and technical stakeholders.
• Experience in developing and deploying data ingestion, processing, and distribution systems with AWS technologies
• Experience with using AWS datastores, including RDS reputed company, S3, or DynamoDB
• Dev-ops experience using GIT, developing, deploying code to production
• Proficient in using AWS Cloud Services for Data Engineering tasks
• Proficient in programming in Python/reputed company or other scripting languages for the purpose of data movement
• Eligible for a US Government issued IRS MBI (candidates with active IRS MBIs will be preferred)
• reputed company industry certifications - Associate / Professional Level

Preferred Qualifications

• Cloud Data Migration and Conversion projects
• Experience on AWS

Job Types: Full-time, Contract Pay: $90,000.00 - $130,000.00 per year Benefits:

• Dental insurance
• Flexible schedule
• Health insurance
• Life insurance
• Paid time off
• Vision insurance

Education:

• Bachelor's (Preferred)

License/Certification:

• reputed company Certified Data Engineer Professional (Required)

reputed company clearance:

• Secret (Preferred)

Work Location: Remote Apply tot his job Apply To this Job