We are looking for a talented and experienced Data Engineer to join our dynamic team. As a Data Engineer, your primary responsibility will be to design, develop, and maintain scalable data solutions that facilitate the seamless generation, collection, and processing of data. Your role will involve creating robust data pipelines, ensuring the integrity and quality of the data, and implementing efficient ETL (Extract, Transform, Load) processes for migrating and deploying data across various systems.

This position requires a deep understanding of data engineering concepts and a strong focus on building high-performance data infrastructure that aligns with the organization’s evolving data needs. You will work closely with other teams to understand business requirements and translate them into reliable and optimized data solutions. As part of the team, you will also play a key role in optimizing the entire data architecture, ensuring the scalability and performance of data systems.

Key Responsibilities

1. Data Pipeline Design & Development. You will be responsible for designing and developing data pipelines that enable smooth and efficient data extraction, transformation, and loading across multiple systems. The pipelines you create will ensure that data is ingested, processed, and moved between various systems in a reliable and timely manner.
2. Data Quality Assurance. One of your core tasks will be to implement processes that maintain the quality and integrity of the data. This includes setting up data validation and cleansing mechanisms to ensure that the data is accurate, clean, and ready for analysis or decision-making.
3. Collaboration with Cross-functional Teams. You will work closely with stakeholders from different departments, including data scientists, analysts, and IT teams, to understand data requirements. Your collaboration will help in the creation of data solutions that are tailored to meet business objectives.
4. Optimization of Data Infrastructure. You will focus on optimizing the data infrastructure for enhanced performance and scalability. By leveraging the latest technologies and best practices, you will ensure that the data systems can handle increasing volumes of data efficiently.
5. Troubleshooting and Issue Resolution. You will be expected to troubleshoot data-related issues and resolve them promptly. This includes providing technical support to stakeholders, debugging problems in data pipelines, and ensuring that the data flow remains uninterrupted.
6. Industry Trend Monitoring. Staying updated with the latest advancements in data engineering technologies is essential. You will proactively learn about new tools and methodologies, integrating them into your work where applicable to improve the team's output.
7. Compliance and Security. Ensuring data security, privacy, and compliance with relevant regulations is a key part of your role. You will be responsible for implementing best practices to safeguard sensitive data and ensure the organization meets legal and regulatory requirements.
8. Mentorship & Knowledge Sharing. As a senior member of the team, you will be expected to mentor junior data engineers. You will help them grow their technical skills and encourage a culture of continuous learning within the team.

Required Professional & Technical Skills

1. Proficiency in PySpark. The ideal candidate should have a strong command over PySpark, with hands-on experience in using it for large-scale data processing and analysis. You will be expected to leverage PySpark’s capabilities for transforming and analyzing big data efficiently.
2. Data Analytics and Machine Learning. A solid understanding of statistical analysis, as well as machine learning algorithms, will be crucial in your role. Experience in applying algorithms such as linear regression, logistic regression, decision trees, and clustering techniques will be beneficial.
3. Data Visualization. Experience with data visualization tools such as Tableau or Power BI is highly desirable. You will use these tools to create visual representations of data that help stakeholders make informed business decisions.
4. Data Munging Techniques. You should have extensive experience in data cleaning, transformation, and normalization. Your ability to handle raw data, remove inconsistencies, and ensure it is in the right format for analysis will be key to maintaining high-quality data.
5. Problem-Solving & Solution-Oriented Mindset. You should be able to identify and resolve complex data-related challenges. This includes finding innovative solutions to optimize the efficiency of data processing pipelines and improve data access across systems.

Qualifications & Experience

1. Minimum 3 Years of Experience. The candidate should have at least 3 years of experience in PySpark and data engineering. This experience should include working on large-scale data processing projects, designing and optimizing data pipelines, and ensuring data quality across systems.
2. Educational Qualification. A minimum of 15 years of full-time education is required, ensuring that the candidate has a strong academic foundation in data-related fields.

Why Join Us?

At our company, you will have the opportunity to work on challenging and exciting projects that have a direct impact on the success of the business. You will be part of a team of highly skilled professionals, and we encourage continuous learning and development. We offer a supportive environment where you can grow your skills and take on greater responsibilities. Additionally, you will have the chance to mentor and share your expertise with junior team members, helping shape the future of data engineering within the organization.

If you are passionate about data engineering, have experience with PySpark, and are ready to take on new challenges, we would love to hear from you! Apply now and be part of a team that is driving data innovation.