As a Principal Software Engineer in our Attribution/Forecasting Product Development team, you’ll be responsible for designing, implementing, and optimizing big data solutions using Scala, Spark, and Hadoop. You will collaborate with cross-functional teams to deploy these solutions both on-premises and in the cloud. Your work will include developing data pipelines, performing data integration, and maintaining workflows for attribution and forecasting engines. Strong communication and Agile methodologies like SCRUM are essential for facilitating global collaboration.

Key Responsibilities

• Develop and optimize large-scale data processing solutions using Scala and Apache Spark.
• Architect and implement robust, scalable systems for both on-prem and cloud environments (AWS, GCP).
• Enhance the performance of Spark jobs, focusing on resource optimization, shuffling, partitioning, and caching.
• Collaborate with cross-functional teams to build, schedule, and maintain workflows.
• Perform data integration and transformation with an in-depth understanding of the Hadoop ecosystem.
• Write efficient SQL queries for handling massive datasets (TBs).
• Mentor junior engineers and drive the team's success through strong leadership.

Qualifications

• 12+ years of experience in Scala and Apache Spark for big data processing.
• Strong expertise in Hadoop (HDFS, YARN, MapReduce).
• Proven experience with Python for developing infrastructure modules.
• Proficiency in SQL for efficient handling of large datasets.
• Hands-on experience with Git and unit/integration test frameworks.
• Ability to work under tight deadlines, prioritize tasks, and collaborate effectively across teams.
• Experience mentoring junior engineers.

Bonus Skills (Nice to Have)

• Hands-on experience with Databricks, Delta Lake, and ELK stack (Elasticsearch, Logstash, Kibana).
• Proficiency in data visualization tools such as Tableau, Power BI, or Grafana.
• Familiarity with Docker and Kubernetes for containerization and orchestration.
• Strong background in analytics and supporting data-driven decisions.

Why Epsilon?

At Epsilon, we believe in fostering a diverse and inclusive environment. We are committed to recruiting and retaining talent from all backgrounds, ensuring a culture that promotes equal opportunity and empowerment.