Azure Data Engineer (Databricks)

Job Title: Azure Data Engineer (Databricks)

Locations: Hybrid (3x a week) - Branchville, NJ | Charlotte, NC | Hartford, CT

Visa Sponsorship: Open to H1B candidates

Job Overview:

We are looking for an Azure Data Engineer with expertise in Databricks, SQL, and Informatica to join a data team supporting multiple projects. The role involves developing data sets for reporting, coding in the Databricks layer, and working closely with business partners. Strong technical problem-solving skills and the ability to work both independently and within a team are crucial.

Key Responsibilities:
• Develop and optimize data sets for reporting using Databricks and SQL
• Independently build and maintain data pipelines in Azure Data Lake
• Work on the migration from Informatica PowerCenter to Databricks
• Collaborate with business partners to understand data needs and reporting requirements
• Support existing data infrastructure and ensure efficient data processing
• Troubleshoot technical issues related to Azure Data Bricks, SQL, and Informatica
• Work closely with offshore teams to coordinate tasks and ensure timely project completion
• Pull data from various sources, including Netezza, policy systems, and external data points

Technical Skills Required:
• Expertise in Azure Databricks
• Strong SQL skills and experience working with Azure Data Lake
• Experience in data engineering and building reporting datasets
• Experience coding in the Databricks layer (Python preferred, Scala is a plus)
• Understanding of data governance and business intelligence best practices
• Informatica PowerCenter experience is a plus
• Experience in insurance/financial services is preferred, but open to other domains
• GitHub experience for version control

Soft Skills:
• Strong communication skills to interact with business users
• Ability to demonstrate the benefits of Databricks over Informatica
• Problem-solving mindset and ability to work independently
• Team player who can collaborate with offshore teams

Day-to-Day Expectations:
• Day 1: Able to troubleshoot and solve technical problems in Databricks
• 30 Days: Understand internal processes, stakeholders, and business use cases
• 60 Days: Start fixing bugs and supporting live projects
• 90 Days: Independently develop data sets for reporting and write Databricks code
• Workload: 70-80% Databricks, 20-30% Informatica and other tools