We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will possess extensive experience in ETL, Data Modeling, and Data Architecture, with proficiency in Scala and hands-on experience in stream data processing using Spark, Kafka, and Spark Structured Streaming.

Key Responsibilities

1. Develop Data Platforms. Build and maintain data platforms including Data Lake, cloud Data Warehouse, APIs, and both batch and streaming data pipelines.
2. Data Processing. Develop batch and stream processing solutions using Apache Spark, Kafka, and Spark Structured Streaming.
3. Orchestration. Utilize Airflow for automating and managing data workflows.
4. Data Transformation. Transform and cleanse raw data using Spark, SQL/PLSQL, and Scala.
5. Data Storage. Implement data storage solutions with Parquet/ORC formats on platforms like PostgreSQL, SQL Server, Teradata, and RDS.
6. Data Modeling. Optimize data storage and retrieval performance through advanced data modeling techniques, including Relational, Dimensional, and E-R modeling.
7. ETL Processes. Maintain data integrity and quality with robust ETL validation and error handling.
8. Deployment Automation. Automate deployment processes with CI/CD tools like Jenkins and Spinnaker.
9. Monitoring & Troubleshooting. Use DataDog and Splunk to monitor and troubleshoot data pipelines, ensuring system reliability.
10. Agile Methodologies. Participate in Agile practices such as Scrum/Kanban, including sprint planning, daily stand-ups, and retrospectives.
11. Code Review. Conduct code reviews to maintain coding standards and best practices.
12. Documentation. Maintain comprehensive documentation of data pipelines, schemas, and processes using Confluence.
13. On-Call Support. Provide on-call support for production data pipelines and resolve issues promptly.
14. Collaboration. Work with cross-functional teams including developers, data scientists, and operations teams to tackle complex data challenges.
15. Continuous Improvement. Stay current with emerging technologies and industry trends to enhance data engineering processes and tools.
16. Reusable Components. Contribute to the development of reusable components and frameworks to streamline data engineering tasks.
17. Version Control. Manage codebase with Git and use IntelliJ IDEA for efficient development and debugging.
18. Security Best Practices. Ensure data security by implementing access controls and handling sensitive data responsibly.

Good-to-Know Skills

1. Programming Languages. Python, Bash/Unix/Linux
2. Big Data Technologies. Hive, Avro, Apache Iceberg, Delta Format
3. Cloud Services. EC2, ECS, S3, SNS, SQS, CloudWatch
4. Databases. DynamoDB, Redis
5. Containerization and Orchestration. Docker, Kubernetes
6. CI/CD Tools. GitHub Copilot
7. Additional Skills. Maven, CLI/SDK

Nice-to-Have Skills

1. Networking. Subnets, Routes
2. Big Data Technologies. Flink

Experience

• Years of Experience. 6-8 years

Skills

1. Primary Skill. Data Engineering
2. Sub Skills. AWS - EKS, AWS - CloudFormation, AWS-Apps, AWS-Infra, AWS DBA
3. Additional Skills. Python, Apache Hive, SQL

How to Apply

If you are passionate about data engineering and want to work in a cutting-edge environment, please submit your resume and a cover letter detailing your experience and qualifications.