We are looking for a talented and driven Data Engineer to join our team. As a Data Engineer, you will play a critical role in the design, development, and maintenance of robust ETL (Extract, Transform, Load) processes that enable seamless data integration from diverse sources. Your expertise in Python and big data technologies will be essential in building efficient and scalable data pipelines that drive our data-driven decisions. In this role, you will be responsible for automating data processing tasks, optimizing performance, and ensuring data quality and integrity through rigorous testing and validation.

Key Responsibilities

ETL Process Design and Development

• Design, develop, and maintain efficient ETL processes to integrate data from multiple sources into centralized data warehouses and lakes.
• Work with business and technical teams to understand data needs and architect scalable and reliable ETL solutions.
• Continuously improve the existing ETL pipelines, optimizing for speed, resource usage, and reliability.

Automation of Data Processing

• Write highly efficient Python scripts to automate repetitive data processing tasks, improving overall productivity and consistency.
• Leverage PySpark and other frameworks to efficiently process large datasets, ensuring high performance even with complex data transformation requirements.

Data Pipeline Development

• Develop and maintain automated data pipelines using cutting-edge tools like Apache Airflow, AWS Glue, and other orchestration tools.
• Manage the entire data lifecycle from extraction to transformation to loading, ensuring timely and accurate data delivery for analytical and operational purposes.

Collaboration with Cross-Functional Teams

• Collaborate with software engineers and data scientists to integrate data solutions into applications, utilizing TypeScript and other relevant technologies.
• Contribute to the design and development of APIs, data integrations, and software solutions that leverage data to enhance business processes and decision-making.

Optimization and Scalability

• Optimize and tune data processing workflows and systems to ensure they meet scalability requirements, handling large volumes of data efficiently.
• Utilize cloud-native technologies and big data tools (such as AWS, Hadoop, Spark, or Kafka) to build highly scalable and fault-tolerant solutions.

Data Quality and Integrity

• Ensure the quality, accuracy, and integrity of the data by implementing comprehensive data validation and testing processes.
• Develop and execute automated tests to continuously monitor the performance and correctness of data pipelines, ensuring that they meet the required business and technical specifications.

Big Data Technologies

• Utilize your knowledge of big data technologies like Hadoop, Spark, and Kafka to process vast amounts of data at scale, ensuring optimal performance and real-time capabilities.
• Stay up-to-date with the latest developments in the big data space, continuously integrating new tools and practices into the data architecture.

Stakeholder Management and Communication

• Collaborate with both technical and non-technical stakeholders to understand their requirements and deliver data solutions that meet their needs.
• Clearly communicate progress, challenges, and solutions to a variety of stakeholders, including team members, leadership, and external partners, through both written and oral reports.

Qualifications

Mandatory Skills

• Strong proficiency in Python and PySpark for building scalable and efficient data processing systems.
• Extensive experience in designing and implementing ETL processes to integrate and transform data from multiple sources.
• Hands-on experience with cloud-based data services and orchestration tools such as Apache Airflow, AWS Glue, or similar.
• Familiarity with big data technologies like Hadoop, Spark, and Kafka is highly desirable.
• Experience in working with large datasets, data warehousing, and data lakes.
• Excellent communication skills, both written and verbal, with the ability to present technical information to non-technical stakeholders.

Nice-to-Have Skills

• Proficiency in TypeScript or other programming languages.
• Familiarity with data visualization tools and reporting systems.
• Previous experience in stakeholder management and cross-team collaboration.
• Experience in building data pipelines in cloud environments (e.g., AWS, GCP, or Azure).

Why Join Us?

This is an exciting opportunity for a motivated data engineer to play a key role in shaping the data architecture of a fast-paced and innovative organization. You will have the chance to work with cutting-edge technologies, collaborate with talented professionals, and contribute to meaningful data-driven solutions. If you are passionate about data engineering and looking for a challenging yet rewarding environment, we would love to hear from you.